Description of problem: A deltaiso from the 15 Alpha TC1 DVD to the 15 Alpha TC2 DVD (either i386 or x86_64) on a system using the new xz compression is expected to be functional, since any changed packages on TC2 should be using the new compression. However, running makedeltaiso and then applydeltaiso, the rebuilt ISO is corrupted in either arch. The corruption happens at 5 specific byte locations and is the same, except for offset, on either arch. [andre@compaq-pc i386]$ cmp Fedora-15-Alpha-i386-DVD.iso new.iso Fedora-15-Alpha-i386-DVD.iso new.iso differ: byte 3341563793, line 12853675 [andre@compaq-pc i386]$ cmp -i 3341563792 -l Fedora-15-Alpha-i386-DVD.iso new.iso 1 22 16 5 43 4 6 270 120 7 207 110 8 54 66 [andre@compaq-pc i386]$ [andre@compaq-pc x86_64]$ cmp Fedora-15-Alpha-x86_64-DVD.iso new.iso Fedora-15-Alpha-x86_64-DVD.iso new.iso differ: byte 3498530625, line 13409740 [andre@compaq-pc x86_64]$ cmp -i 3498530624 -l Fedora-15-Alpha-x86_64-DVD.iso new.iso 1 22 16 5 43 4 6 270 120 7 207 110 8 54 66 [andre@compaq-pc x86_64]$ I manually identified all packages on each TC2 ISO that still use the old compression and verified that the identical packages exist on TC1, so none of them should generate a deltarpm. I also confirmed that the generated deltaiso and the rebuilt corrupted ISO are identical regardless of whether I create them on a F14 system modified to use the new compression (see http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ) or on an updated Rawhide VM. I also checked the TC1 and TC2 checksums, both before and after. All this makes it unlikely that either my hardware or the use of a modified F14 system is the problem. It could be very helpful if someone who knows how to examine the content of ISOs could find out what is at the locations listed above. Version-Release number of selected component (if applicable): deltarpm-3.6-0.5.20110121git.fc15 How reproducible: always Steps to Reproduce: 1. Download TC1 and TC2 DVD ISOs of the same arch. 2. Use either a modified F14 system (see above) or a F15/Rawhide system to run makedeltaiso and then applydeltaiso. Compare the original newiso with the rebuilt one using cmp as above. Actual results: Corruption in the specific 5 byte locations listed above Expected results: ISOs should match Additional info:
Should have also mentioned that on either arch, newiso and the rebuilt corrupted ISO are exactly the same size, the only difference are the 5 changed bytes listed above. On i386, both are 3474040832 bytes. On x86_64, both are 3630661632 bytes. The sizes and MD5 sums of the disos are i386: 491276437 bytes, 989b840b7fc1f520a54d5b2354aa9ab9 x86_64: 619559697 bytes, 409dda0c7fc7bc3e3bff16a21cc1a037
I loop mounted good and bad ISOs for each arch and did a recursive diff. In each case, the corruption was in the middle of the xorg-x11-drivers-7.4-2.fc15 RPM. [andre@compaq-pc tmp]$ cmp foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm differ: byte 8081, line 26 [andre@compaq-pc tmp]$ cmp -i 8080 -l foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm 1 22 16 5 43 4 6 270 120 7 207 110 8 54 66 [andre@compaq-pc tmp]$ [andre@compaq-pc tmp]$ cmp foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm differ: byte 8001, line 28 [andre@compaq-pc tmp]$ cmp -i 8000 -l foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm 1 22 16 5 43 4 6 270 120 7 207 110 8 54 66 [andre@compaq-pc tmp]$ This package uses the new compression in TC2. I was able to generate a working deltarpm between the old and new RPMs using {make,apply}deltarpm with the -r option, so have no idea why the copy in the rebuilt ISO is corrupted.
The context is (the hexdump starts at 3498530554) 00002f0 3d65 6567 656e 6972 0063 7063 6f69 7800 00002f0 e = g e n e r i c \0 c p i o \0 x 0000300 007a 0032 3878 5f36 3436 722d 6465 6168 0000300 z \0 2 \0 x 8 6 _ 6 4 - r e d h a 0000310 2d74 696c 756e 2d78 6e67 0075 0000 0800 0000310 t - l i n u x - g n u \0 \0 \0 \0 \b 0000320 0000 3f00 0000 0700 ffff e0fd 0000 1000 0000320 \0 \0 \0 ? \0 \0 \0 \a 377 377 375 340 \0 \0 \0 020 0000330 37fd 587a 005a 0a00 fbe1 a10c 0002 0121 0000330 375 7 z X Z \0 \0 \n 341 373 \f 241 002 \0 ! 001 0000340 0012 0000 b823 2c87 00e0 007b 5d1d 1800 0000340 022 \0 \0 \0 # 270 207 , 340 \0 { \0 035 ] \0 030 where the original bytes are on the last line, so it looks to be near the start of an xz compression block.
If you rpm2cpio the two different versions of the rpm, is the uncompressed data identical?
Ok, scratch that last question. Do you have a copy of the old xorg-x11-drivers rpm anywhere? Can you give me a link?
Forget it, I checked in koji and it was easy to see what the previous one was. I made a deltarpm between xorg-x11-drivers-7.4-1.fc14 and xorg-x11-drivers-7.4-2.fc15, then applied at using -r and got an md5 mismatch. So it is a compression format change problem, but I don't know why it's showing up as xorg-x11-drivers-7.4-2.fc15 was built after we should have had updated versions of xz on our builders. Maybe it would be a good idea to ask for another rebuild of xorg-x11-drivers and see what that looks like.
I think the differences are the xz dictionary size parameter (1MiB verses 256KiB) for the LZMA2 filter and a 32-bit CRC. The package is small enough for this difference not to matter in terms of uncompressing.
That sounds about right. I was able to manually match the original xz output by using xz -4 rather than rpm's default level 3. The question is, why was this rpm compressed using a different setting. Deltarpm (and therefore deltaiso) *depends* on being able to recreate a byte-for-byte copy of the original compressed rpm, and that depends on compression being deterministic.
Is deltarpm being too clever here? The file archive in xorg-x11-drivers are the same in both versions (both are actually empty), so could deltarpm be deciding to reuse the existing compressed cpio file rather than creating a new one. It would make sense for deltarpm to do this in most cases because it would avoid uncompressing and recompressing the contents only to end up with what you started with, and it would only be a problem if something changed in the compression algorithm.
(In reply to comment #6) > Forget it, I checked in koji and it was easy to see what the previous one was. > > I made a deltarpm between xorg-x11-drivers-7.4-1.fc14 and > xorg-x11-drivers-7.4-2.fc15, then applied at using -r and got an md5 mismatch. See the end of comment 2 - I previously copied xorg-x11-drivers-7.4-1.fc14 out of the loop-mounted TC1 ISO, xorg-x11-drivers-7.4-2.fc15 out of the loop-mounted TC2 ISO, then used {make,apply}deltarpm with the -r option and got a successful rebuild, with my F14 box modified to use the new compression as indicated in my original post. This worked on both i386 and x86_64. Were you using the old compression?
Re: comment #10: Aw crumbs, yeah, I'm doing this on an F14 box, so, yeah, of course it won't work. /me smacks my forehead. Re: comment #9: You know, I think you might be right. As far as I know, makedeltarpm *always* recreates the payload, but I noticed makedeltaiso saying that xorg-x11-drivers.i686 is "unchanged", so it may be reusing the compressed payload from xorg-x11-drivers-7.4-1.fc14. Of course, given deltarpm's codebase, this will be a royal pain to fix. Let me see what I can do.
In case it matters, the 4 RPMs on TC1 and TC2 are all signed and are identical to http://download.fedora.redhat.com/pub/fedora/linux/releases/14/Fedora/i386/os/Packages/xorg-x11-drivers-7.4-1.fc14.i686.rpm http://download.fedora.redhat.com/pub/fedora/linux/releases/14/Fedora/x86_64/os/Packages/xorg-x11-drivers-7.4-1.fc14.x86_64.rpm http://download.fedora.redhat.com/pub/fedora/linux/development/15/i386/os/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm
I've just checked the code, and, yes, makedeltaiso basically skips over rpms whose *uncompressed* payloads haven't changed. Unfortunately, I'm hesitant to screw around with this code, as the easiest alternative is to have makedeltaiso delta *every* rpm on the DVD. This would obviously take *much* longer and leave larger deltaisos to boot, and is therefore a nonstarter. A better alternative would be to check whether the *compressed* data has changed after checking the uncompressed data, but this would either be time-consuming or a major code restructure (or possibly both). Let me make contact with Michael S. upstream and see if he has any ideas. Andre, how important is this bug? We won't run into this problem with the next ISO (TC2->TC3), and the only reason we hit it this time is that we've changed the compression settings.
I don't understand what's different between TC1->TC2 and TC2->RC1 (which is scheduled for tomorrow, unless it's delayed). Is it that this bug only happens due to this particular xorg-x11-drivers update? And does that mean that all deltas from F14 Final to later ISOs will be broken as well, even when all packages in the new ISO are using the new compression? (Although if that's the case, a fixed deltarpm wouldn't help much, since it would depend on RPM 4.9 making the F14 modification impractical. The version I used was an older one that still depended on RPM 4.8.) Also, do you know what the situation is regarding future xz compression changes? Is it pretty much set in stone now, or still subject to change? Given the large potential savings from using delta compression on ISOs, it's a shame that all this can't be made to Just Work somehow (from the user's standpoint). Even going from (N-1) Final to N Final saves about 50% but if the compression changes every other year, that means one out of every 4 such deltas would be potentially unusable (since even without the current bug, there may be no practical way to modify an (N-1) system to use N's compression). BTW, I tried pinging you several times on #fedora-qa.
I wonder if you could have a more general "treat as new" list of packages where if a package was on the list the delta iso program just ignored the old version and included the full rpm, as if it were a new package rather than an update. It would allow you to get around problems like this without needing to change the code each time, and without extra processing in general. It might be possible to do a one-off hack along these lines to generate a TC1->TC2 delta iso that worked. Comment 14: TC2->RC1 should work unless any other similar "empty" packages have been updated between these two milestones. But you will have the same problem with F14->F15 (unless you do something sneaky and include a specially modified intermediate ISO that omits the xorg-x11-drivers package - your updates from F13 to F14 include several steps anyway so you could generate the first step, rebuild that iso without the xorg-x11-drivers package regenerate the diff to the modified iso, and then generate the diffs from the modified iso to the final iso as normal).
I've sent off an email to upstream asking about a fix. In the meantime, Michael, your idea in comment 15 should work for TC1 -> TC2 as well (assuming it works at all; I'm not sure if applydeltaiso checks checksums on the old ISO).
(In reply to comment #15) > It might be possible to do a one-off hack along these lines to generate a > TC1->TC2 delta iso that worked. I decided to try doing a one-off hack and the results are in koji at http://koji.fedoraproject.org/koji/taskinfo?taskID=2844891 I tried it on a pair of mini isos from the fc14 xorg-x11-drivers package and another rpm to the fc15 xorg-x11-drivers package and an rpm update, and it worked, so I expect it will generate a TC1->TC2 deltaiso that will work with the normal applydeltaiso program, though no promises. It might do FC14->TC2 or later as well though that depends on what else has changed. It is a scratch build so it will be automatically deleted after about 2 weeks.
(In reply to comment #16) > In the meantime, Michael, your idea in comment 15 should work for TC1 -> TC2 as > well (assuming it works at all; I'm not sure if applydeltaiso checks checksums > on the old ISO). applydeltaiso doesn't run a checksum on the source iso, but is depends closely on the contents so almost any difference in the source iso is likely to cause it to fail (the only thing I found that worked was changing a byte in the xorg-x11-drivers rpm which was ignored by the delta iso I generated with the hacked makedeltaiso). What I meant in comment 15 was that you use two (or more) delta isos to get from the start iso to the end one, where the first intermediate iso is constructed to omit the xorg-x11-drivers rpm. If there are going to be intermediate isos anyway (as might be the case going from F14 to F15) it makes sense for this to be close to what would have been the first intermediate step.
(In reply to comment #13) > A better alternative would be to check whether the *compressed* data has > changed after checking the uncompressed data, but this would either be > time-consuming or a major code restructure (or possibly both). Let me make > contact with Michael S. upstream and see if he has any ideas. I don't think makedeltaiso's speed is too important compared to being reliable in terms of generating working deltas. Applydeltaiso's speed is far more important. Anyway, I don't understand why it compares old and new rpms by comparing their uncompressed data. A direct comparison would be 100% reliable - wouldn't it also be faster as well? (I'm probably missing something so excuse me if this is a dumb question.)
qemu-kvm-0.13.0-0.5.20100809git25fdf4a.fc15 is on TC2, contains no files, and is still using the old compression. so it appears this will happen again once.
Created attachment 479629 [details] Possible patch to compare compressed data Here is another attempt at a fix, along the lines of comment #13 . It does what I would expect on my rather minimal test example. I have done a scratch build with the patch applied at http://koji.fedoraproject.org/koji/taskinfo?taskID=2850397 if you want to test it.
Your patch works! I used it in a F15 VM (I can't update F14 to it since it depends on rpm-4.9) and makedeltaiso to generate the i386 TC1->TC2 DVD diso. The only difference in the screen output was that the xorg-x11-drivers line now says "creating delta" instead of "unchanged". The new diso works with applydeltaiso from the unpatched version. It's hard to tell whether makedeltaiso is slower due to having to run it in a VM. I'll make and verify the x86_64 TC1->TC2 DVD diso as well and post both. Jonathan, if this patch looks good, could you push an update, and maybe a custom build that only depends on rpm-4.8 as well (so it can be used to update the F14 version)?
Created attachment 479663 [details] Attempt two on the patch to compare compressed data The patch looks good to me. The one thing I don't like is that we read in the new rpm both compressed and uncompressed even if it's byte-for-byte identical. I'm assuming you were worried about namebuf[0] being set, but we set it to 254 anyway, so that shouldn't be a problem. I've modified the patch so it doesn't uncompress (or reopen) anything if the compressed payloads are identical. I've done a local test with a single-rpm iso and it works correctly, but, Andre, I'd appreciate it if you could test with TC1->TC2. And, yes, users won't need an updated applydeltaiso. Michael, many thanks for the patch.
I attempted to build the 64-bit TC1->TC2 delta using Michael's patched version on my 64-bit F15 VM. The 32-bit delta build worked, but this time it used lots of CPU and I/O but didn't progress beyond /tmp/foo/F15-Alpha.TC1/x86_64/Fedora-15-Alpha-x86_64-DVD.iso: 2920 rpms Fedora-15-Alpha-x86_64-DVD.iso: 2925 rpms reading old iso (omitting payloads) size without payloads is 409533929 bytes reading new iso (omitting payloads) size without payloads is 411574724 bytes creating iso diff after almost an hour. I don't know if this would take extremely long due to my VM only having 1 GiB of RAM and being slower. Please do a build of the new patch and I'll try it. Thanks.
http://koji.fedoraproject.org/koji/taskinfo?taskID=2851029 It's built against dist-f15
(In reply to comment #23) > I've modified the patch so it doesn't uncompress (or reopen) anything if the > compressed payloads are identical. I've done a local test with a single-rpm > iso and it works correctly, but, Andre, I'd appreciate it if you could test > with TC1->TC2. Yes, I had assumed that payread might set more than the first character of namebuf without thinking to check. (In reply to comment #22) > It's hard to tell whether > makedeltaiso is slower due to having to run it in a VM. I would expect it to be marginally slower because in most cases it is doing extra reads and memory moves, but most of the time is probably used in uncompressing and compressing contents which is unchanged (except for any rpms like xorg-x11-drivers where we do save some processing but those will be rare). I have tested Jonathan's revised SRPM rebuilt for F14 (scratch build at http://koji.fedoraproject.org/koji/taskinfo?taskID=2851048 ) on my test case and it works for me too.
I'm not able to use Jonathan's version either in a VM. I downgraded to the original F15 version and the same problem happens with that, so the VM must just be resource-starved. Jonathan: could you create a build that depends on rpm-4.8, but uses the new compression, so I could do the F14 modification as in http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ? It seems to be the only way to do the testing in a reasonable amount of time. Thanks.
(In reply to comment #27) > I'm not able to use Jonathan's version either in a VM. I downgraded to the > original F15 version and the same problem happens with that, so the VM must > just be resource-starved. Jonathan: could you create a build that depends on > rpm-4.8, but uses the new compression, so I could do the F14 modification as in > http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ? It > seems to be the only way to do the testing in a reasonable amount of time. > Thanks. Well it isn't the only way. If your setup allows it you could mount the file systems from your VM on the parent box and run makedeltaiso under a chroot (it would see the files of the VM but with the memory of the parent box). However I have also tried a rebuild of Jonathan's SRPM on F14 against the later xz version and uploaded the resulting rpms (except the debuginfo one) to http://myoung.fedorapeople.org/temp/ and I think it should do what you want. Incidentally I don't think the dependency on rpm-4.8 actually matters for makedeltaiso (though it might for applydeltaiso) so the patched F15 executable may work.
Jonathan explained to me on IRC how to build his source RPM on my F14 box, which I did, then installed it. Then I ran {make,apply}deltaiso for TC1->TC2 x86_64. The result was very strange: it claimed the ISO was corrupt, but in fact it was good. It turned out that makedeltaiso was storing the wrong md5 sum: iso diff done, final compression: 17.1% iso md5sum is: 971cc87e295b172010c3a98be0792c16 when in fact the correct md5 sum for TC2 x86_64 is ac0ea9a27177d1ef44ab4c16524b1bbf which is what the unpatched makedeltaiso reports.
Ok, I see the problem. Most of the MD5 checksum comes from reading the uncompressed data, so, Michael, you were right, we do need to read in the uncompressed data no matter what. Ugh.
Compiling the src rpm from http://koji.fedoraproject.org/koji/taskinfo?taskID=2850397 on my F14 box with the new xz compression and using the resulting binary rpms for both makedeltaiso and applydeltaiso, it works properly for TC1->TC2, both i386 and x86_64. Running both on bare metal means I can do a direct speed comparison of unpatched vs. patched. If anything, the patched version is a little faster. [andre@compaq-pc ~]$ diff makedeltaiso_TC2_i386.txt makedeltaiso_TC2_i386_patched.txt 2840c2840 < xorg-x11-drivers.i686: unchanged...13.6% --- > xorg-x11-drivers.i686 (xz.2): creating delta...13.6% 2941,2943c2941,2943 < real 31m11.871s < user 25m14.049s < sys 1m49.963s --- > real 29m29.843s > user 24m12.433s > sys 2m14.964s [andre@compaq-pc ~]$ diff makedeltaiso_TC2_x86_64.txt makedeltaiso_TC2_x86_64_patched.txt 2838c2838 < xorg-x11-drivers.x86_64: unchanged...16.7% --- > xorg-x11-drivers.x86_64 (xz.2): creating delta...16.7% 2937,2939c2937,2939 < real 33m30.262s < user 28m25.854s < sys 2m22.838s --- > real 32m48.911s > user 27m24.683s > sys 1m27.068s [andre@compaq-pc ~]$
The qemu-kvm package was not updated yet in RC1, and as expected the TC2->RC1 deltas generated by the unpatched and patched versions of deltarpm are identical and both work. (See comment 20.) I posted both them and the working TC1->TC2 deltas on my download page ( https://fedoraproject.org/wiki/User:Robatino/Downloads ).
Jonathan: could you push fixed versions of deltarpm before Michael's scratch build in comment 21 from 2/18 expires (in case someone else needs the patch - I've already saved a copy of the src rpm)? Thanks.
deltarpm-3.6-0.6.20110223git.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc14
deltarpm-3.6-0.6.20110223git.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc15
deltarpm-3.6-0.6.20110223git.fc15 has been pushed to the Fedora 15 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update deltarpm'. You can provide feedback for this update here: https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc15
deltarpm-3.6-0.6.20110223git.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.
deltarpm-3.6-0.6.20110223git.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.