Description of problem: A bad thunderbird-24.5.0-1.fc19.i686 rpm got pushed to the mirrors. yum tries to install it: error: unpacking of archive failed on file /usr/lib/thunderbird/langpacks/langpack-si.org.xpi;53660a70: cpio: read error: thunderbird-24.5.0-1.fc19.i686: install failed error: thunderbird-24.4.0-1.fc19.i686: erase skipped thunderbird-24.5.0-1.fc19.i686 was supposed to be installed but is not! thunderbird-24.4.0-1.fc19.i686 was supposed to be removed but is not! It shouldn't. # rpm -K /data/repos/fedora/updates/19/i386/thunderbird-24.5.0-1.fc19.i686.rpm /data/repos/fedora/updates/19/i386/thunderbird-24.5.0-1.fc19.i686.rpm: RSA sha1 (MD5) PGP MD5 NOT OK Version-Release number of selected component (if applicable): yum-3.4.3-132.fc19.noarch rpm-4.11.2-2.fc19.i686
In my opinion, it is a BIG SECURITY RISK. Yum shouldn't install corrupted files. It should detect the bad MD5 checksum, and should detect the incorrect public/private key calculation. See https://bugzilla.redhat.com/show_bug.cgi?id=1093927 The priority and severity of this bug ( Bug 1094846 ) should be VERY HIGH, since it compromises the security of the system, and is a door to attacks. How has this happened? It should be investigated and fixed quickly.
Yum (and by default, rpm too) only verify the header signatures. The payload contents are covered by SHA256 digests on individual files, which are contained in the signed header so a corrupted payload will get noticed and install of such a package aborted. A corrupted file from the payload is only ever visible with a temporary name (eg /some/path;52b144bb) while unpacking and moved to place after verifying. Yum *could* be made to the header+payload signature as well, but that comes at quite a runtime cost.
(In reply to Panu Matilainen from comment #2) > A corrupted file from the payload is only > ever visible with a temporary name (eg /some/path;52b144bb) while unpacking > and moved to place after verifying. Ehm, that ends up being quite misleading. A file is only moved to place after it passes digest verification, so a corrupted file never gets moved to place.
FWIW - In this case it rendered thunderbird unusable because it tried to install an incomplete package and failed. I'm not sure I've ever heard of this happening before, but it does seem to be a gap in integrity checking.
Yes you can end up with a broken installation because rpm moves the files to final destination one by one during unpacking, so when there is a failure you can end up with files from two different versions on disk. Which might or might not work, and clearly this is not exactly an optimal situation. The point I'm making is that the issue is far more subtle than the "security catastrophe, yum fails to detect corrupted packages" implied in comment #1.
Created attachment 895809 [details] Applicable corrupted thunderbird archive I have attached the corrupted thunderbird archive for reference. Note that yum partially installed this rpm resulting in an unusable thunderbird. $ rpm -K thunderbird-24.5.0-1.fc19.i686.rpm.defective thunderbird-24.5.0-1.fc19.i686.rpm.defective: rsa sha1 (MD5) PGP MD5 NOT OK
(In reply to Panu Matilainen from comment #3) > (In reply to Panu Matilainen from comment #2) > > A corrupted file from the payload is only > > ever visible with a temporary name (eg /some/path;52b144bb) while unpacking > > and moved to place after verifying. > > Ehm, that ends up being quite misleading. A file is only moved to place > after it passes digest verification, so a corrupted file never gets moved to > place. What is performing the digest verification? Based on my understanding of the error message from yum: error: unpacking of archive failed on file /usr/lib/thunderbird/langpacks/langpack-si.org.xpi;53660a70: cpio: read The unpacking aborted to a cpio read error not a digest verification error. Please correct me if I am wrong.
There's no digest to verify because its failing before all data was read, ie its known broken already.
(In reply to Panu Matilainen from comment #8) > There's no digest to verify because its failing before all data was read, ie > its known broken already. Thanks for your reply. If I understand you correctly, no digest verification occurs until after a file is extracted from the rpm package. The problem is that yum does not detect when the rpm package itself is truncated or corrupted' This results in files being extracted and moved from a truncated or corrupted rpm causing an unusable application. However, time rpm -K thunderbird-24.5.0-1.fc19.i686.rpm.defective thunderbird-24.5.0-1.fc19.i686.rpm.defective: rsa sha1 (MD5) PGP MD5 NOT OK real 0m0.894s user 0m0.735s sys 0m0.104s does detect a problem with the rpm package and does not take very long on my Intel Atom N270. Therefore, it is unclear to me why yum would not perform an rpm -K type check on a package before any extraction occurs.
(In reply to Orion Poplawski from comment #0) > Description of problem: > > A bad thunderbird-24.5.0-1.fc19.i686 rpm got pushed to the mirrors. yum > tries to install it: > > error: unpacking of archive failed on file > /usr/lib/thunderbird/langpacks/langpack-si.org.xpi; > 53660a70: cpio: read > error: thunderbird-24.5.0-1.fc19.i686: install failed > error: thunderbird-24.4.0-1.fc19.i686: erase skipped > thunderbird-24.5.0-1.fc19.i686 was supposed to be installed but is not! > thunderbird-24.4.0-1.fc19.i686 was supposed to be removed but is not! > > It shouldn't. > > # rpm -K > /data/repos/fedora/updates/19/i386/thunderbird-24.5.0-1.fc19.i686.rpm > /data/repos/fedora/updates/19/i386/thunderbird-24.5.0-1.fc19.i686.rpm: RSA > sha1 (MD5) PGP MD5 NOT OK > > Version-Release number of selected component (if applicable): > yum-3.4.3-132.fc19.noarch > rpm-4.11.2-2.fc19.i686 Just to re-iterate the seriousness of this situation. It's one thing for yum to partially install a corrupted/truncated thunderbird rpm and as a result, you are not able to use thunderbird for a few days to read your e-mail until the repositories are corrected. What if this had happened with a kernel or a core X windows package which then rendered the system un-bootable or unable to use any graphical applications?
I think vernonjvs is right (comment 9 and comment 10). And I would like to add something more. To install a rpm, the following steps should be done: 1. check authenticity 2. check integrity 3. uninstall old rpm 4. install new rpm More details for steps 1 and 2: 1. check authenticity: Calculate checksum of the rpm (or the checksum of the header, according to comment 2). And check that the calculated checksum is equal to the result of applying the fedora public key to the encrypted checksum (the encrypted checksum is into the rpm and was calculated with the fedora private key at fedora servers). If they are equal, then the authenticity is ok (the rpm is an authentic fedora file). 2. check integrity: According to comment 2, the header (already checked at step 1) contains a checksum for each file of the payload. So, to check integrity, a checksum should be calculated for every file in the payload, and see if the resulting checksums are equal to the checksums at the header. If they are equal, then integrity is ok (the rpm has not been modified). From the discussion, I understand that step 1 (authenticity) was ok, and that some file was modified in the payload, so step 2 (integrity) was NOT OK. So steps 3 and 4 shouldn't have been done. So here questions come: Why was the old rpm uninstalled, and the new rpm started to be installed? (please change status from "new" to "confirmed")
IMHO, if the process of updating a package is so critical that aborting it halfway breaks that package, then it should not start until a reasonable assurance it will go to its end can be asserted. If this assurance requires that the integrity of ALL files is checked OK before ANY single file from the update is installed, then this check should be done, and there is no discussion about it. For me the issue of extra computing time to perform this check is not a valid reason to avoid it, considering the time lost by the end user due to an broken package with no fallback (which is not measured in terms of few seconds of computing time, but in hours...). Additional point of view: all this topic (and all related bug reports) started because some users (including me) complained about a broken thunderbird. In other words, it seems that the only way to detect that a corrupted package was sent to mirrors is just that some users complain about the results of installing it on their Fedora box, breaking some packages. What about QA? No reason seems to be given so far WHY/HOW a correct package on koji (the master server if I understand) ends up in an incorrect copy on some mirrors. So the process of duplicating packages on mirrors needs *at least* a serious review: this bug demonstrates that this process (as a whole) is not as failsafe (strong? secure?) as it can be expected to be. As mentioned in Comment 10, what would be the result, should this issue happen to a vital package such as a kernel update? In such a case I know it is still possible to boot from the N-1 kernel release by selecting it at boot time, but this is tedious; and for some almost-necessary packages like the X server, there is simply not such fallback at all (so no graphical apps can be run, including the browser itself to fetch for some help...). What for instance about such an issue on an SELinux policy update, or on a security patch update an a package, failing but yielding in a still workable package but missing the security patch? In both cases the user can believe he/she is protected but in fact not really.
As amreg has explained (comment 12), this bug has discovered a BIG SEGURITY RISK in yum: If integrity is not checked before uninstallation/installation, then an evil rpm could be installed. For example, someone could take an original fedora rpm, modify the payload to add evil software, and inject the rpm into the fedora mirror network, replacing the original rpm by the evil rpm. Yum clients will download the rpm, will check the authenticity of the header, and WON'T check the integrity of the payload, so the rpm will be installed, and the evil code will be run. I repeat, it is a BIG SEGURITY RISK that should be fixed quickly.
(In reply to Fdor from comment #13) > As amreg has explained (comment 12), this bug has discovered a BIG SEGURITY > RISK in yum: If integrity is not checked before uninstallation/installation, > then an evil rpm could be installed. No. Although I'm somewhat confused here (esp. by Panu's response), I suspect you are more confused. > For example, someone could take an original fedora rpm, modify the payload > to add evil software, and inject the rpm into the fedora mirror network, > replacing the original rpm by the evil rpm. Yum clients will download the > rpm, will check the authenticity of the header, and WON'T check the > integrity of the payload, so the rpm will be installed, and the evil code > will be run. No. So yum does this, roughly, for Fedora/updates: 1. HTTPS connection to get metalink. 2. extract checksum for repo metadata. 3. download repo metadata, and verify against #2. 4. extract package checksums from #3. 5. download packages, and verify against #4. 6. if gpg-verify is set, verify packages against GPG keys. 7. install packages. ...Panu seems to be implying that the check at #6 can pass even though rpm -K says there is a problem (I think) ... however even if true, you'd still need access to the Fedora rel-eng machine to get a bad package past #1-#5.
(In reply to James Antill from comment #14) > (In reply to Fdor from comment #13) > > As amreg has explained (comment 12), this bug has discovered a BIG SEGURITY > > RISK in yum: If integrity is not checked before uninstallation/installation, > > then an evil rpm could be installed. > > No. Although I'm somewhat confused here (esp. by Panu's response), I > suspect you are more confused. > > > For example, someone could take an original fedora rpm, modify the payload > > to add evil software, and inject the rpm into the fedora mirror network, > > replacing the original rpm by the evil rpm. Yum clients will download the > > rpm, will check the authenticity of the header, and WON'T check the > > integrity of the payload, so the rpm will be installed, and the evil code > > will be run. > > No. So yum does this, roughly, for Fedora/updates: > > 1. HTTPS connection to get metalink. > > 2. extract checksum for repo metadata. > > 3. download repo metadata, and verify against #2. > > 4. extract package checksums from #3. > > 5. download packages, and verify against #4. > > 6. if gpg-verify is set, verify packages against GPG keys. > > 7. install packages. > > ...Panu seems to be implying that the check at #6 can pass even though rpm > -K says there is a problem (I think) ... however even if true, you'd still > need access to the Fedora rel-eng machine to get a bad package past #1-#5. I have previously attached the actual defective or perhaps truncated thunderbird package which was in the repositories and renamed it to thunderbird-24.5.0-1.fc19.i686.rpm.defective. Rather than debate what can and can not happen, please download thunderbird-24.5.0-1.fc19.i686.rpm.defective and try to install it with yum. You will find that yum partially installs thunderbird and then aborts. When upgrading from a previous version of thunderbird, this leaves a broken thunderbird installation since there are now some files with a version mismatch. Since rpm -K can detect that thunderbird-24.5.0-1.fc19.i686.rpm.defective is a bad package, yum should be able to detect it as well and not try to install it. Additional details on the history can be found https://bugzilla.redhat.com/show_bug.cgi?id=1093927
(In reply to James Antill from comment #14) According to the steps you have written, it is impossible for an evil/corrupted rpm to be downloaded and installed, even if it is injected into the fedora mirror network. But, as comment #15 has explained, it HAS HAPPENED: - a corrupted package has been injected into the fedora mirror network. - yum clients have downloaded the rpm. [steps 1 to 5] - yum clients have installed the rpm partially. [steps 6 to 7] If it is impossible, how has it happened?
Well, after nearly 2 months (bug 1093927 was posted at 2014-05-03), we still have no official explanation on WHAT HAPPENED. And it looks like it is not going to be investigated. At https://fedorahosted.org/rel-eng/ticket/5898 they only say that createrepo should have detected the corrupted rpm. So what are they going to do? Are they going to analize createrepo? The original bug (bug 1093927) was closed, and the only conclusion I get from that page is that the problem was related to the mirror infrastructure. So are they going to review the mirror infrastructure? And the current bug (bug 1094846) says that the thing that happened can never happen! Perhaps I'm missing something but, sincerely, I can't understand this way of working. So, as a last try, I have reported the bug 1114208 to include a "rpm -K <rpm>" command in yum, to detect corrupted rpm files.
This message is a notice that Fedora 19 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 19. It is Fedora's policy to close all bug reports from releases that are no longer maintained. Approximately 4 (four) weeks from now this bug will be closed as EOL if it remains open with a Fedora 'version' of '19'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 19 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
(In reply to James Antill from comment #14) > (In reply to Fdor from comment #13) > No. So yum does this, roughly, for Fedora/updates: > > 1. HTTPS connection to get metalink. > > 2. extract checksum for repo metadata. > > 3. download repo metadata, and verify against #2. > > 4. extract package checksums from #3. > > 5. download packages, and verify against #4. > > 6. if gpg-verify is set, verify packages against GPG keys. > > 7. install packages. > > ...Panu seems to be implying that the check at #6 can pass even though rpm > -K says there is a problem (I think) ... however even if true, you'd still > need access to the Fedora rel-eng machine to get a bad package past #1-#5. Yes, and yes. Also this is at least somewhat mitigated by the recent security update, which I guess went out for F19 as well.
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.