Bug 677578 - unexpected corruption in deltaiso-built F15 Alpha TC2 DVD image (either i386 or x86_64)
Summary: unexpected corruption in deltaiso-built F15 Alpha TC2 DVD image (either i386 ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: deltarpm
Version: rawhide
Hardware: All
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Jonathan Dieter
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-02-15 09:48 UTC by Andre Robatino
Modified: 2011-03-03 03:36 UTC (History)
2 users (show)

Fixed In Version: deltarpm-3.6-0.6.20110223git.fc15
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-02-26 03:59:47 UTC


Attachments (Terms of Use)
Possible patch to compare compressed data (2.99 KB, patch)
2011-02-18 23:20 UTC, Michael Young
no flags Details | Diff
Attempt two on the patch to compare compressed data (3.48 KB, patch)
2011-02-19 10:53 UTC, Jonathan Dieter
no flags Details | Diff

Description Andre Robatino 2011-02-15 09:48:56 UTC
Description of problem:
A deltaiso from the 15 Alpha TC1 DVD to the 15 Alpha TC2 DVD (either i386 or x86_64) on a system using the new xz compression is expected to be functional, since any changed packages on TC2 should be using the new compression. However, running makedeltaiso and then applydeltaiso, the rebuilt ISO is corrupted in either arch. The corruption happens at 5 specific byte locations and is the same, except for offset, on either arch.

[andre@compaq-pc i386]$ cmp Fedora-15-Alpha-i386-DVD.iso new.iso
Fedora-15-Alpha-i386-DVD.iso new.iso differ: byte 3341563793, line 12853675
[andre@compaq-pc i386]$ cmp -i 3341563792 -l Fedora-15-Alpha-i386-DVD.iso new.iso
        1  22  16
        5  43   4
        6 270 120
        7 207 110
        8  54  66
[andre@compaq-pc i386]$

[andre@compaq-pc x86_64]$ cmp Fedora-15-Alpha-x86_64-DVD.iso new.iso
Fedora-15-Alpha-x86_64-DVD.iso new.iso differ: byte 3498530625, line 13409740
[andre@compaq-pc x86_64]$ cmp -i 3498530624 -l Fedora-15-Alpha-x86_64-DVD.iso new.iso
        1  22  16
        5  43   4
        6 270 120
        7 207 110
        8  54  66
[andre@compaq-pc x86_64]$

I manually identified all packages on each TC2 ISO that still use the old compression and verified that the identical packages exist on TC1, so none of them should generate a deltarpm. I also confirmed that the generated deltaiso and the rebuilt corrupted ISO are identical regardless of whether I create them on a F14 system modified to use the new compression (see http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ) or on an updated Rawhide VM. I also checked the TC1 and TC2 checksums, both before and after. All this makes it unlikely that either my hardware or the use of a modified F14 system is the problem.

It could be very helpful if someone who knows how to examine the content of ISOs could find out what is at the locations listed above.


Version-Release number of selected component (if applicable):
deltarpm-3.6-0.5.20110121git.fc15

How reproducible:
always

Steps to Reproduce:
1. Download TC1 and TC2 DVD ISOs of the same arch.
2. Use either a modified F14 system (see above) or a F15/Rawhide system to run makedeltaiso and then applydeltaiso. Compare the original newiso with the rebuilt one using cmp as above.
  
Actual results:
Corruption in the specific 5 byte locations listed above

Expected results:
ISOs should match

Additional info:

Comment 1 Andre Robatino 2011-02-15 10:04:36 UTC
Should have also mentioned that on either arch, newiso and the rebuilt corrupted ISO are exactly the same size, the only difference are the 5 changed bytes listed above. On i386, both are 3474040832 bytes. On x86_64, both are 3630661632 bytes. The sizes and MD5 sums of the disos are

i386: 491276437 bytes, 989b840b7fc1f520a54d5b2354aa9ab9
x86_64: 619559697 bytes, 409dda0c7fc7bc3e3bff16a21cc1a037

Comment 2 Andre Robatino 2011-02-15 16:45:03 UTC
I loop mounted good and bad ISOs for each arch and did a recursive diff. In each case, the corruption was in the middle of the xorg-x11-drivers-7.4-2.fc15 RPM.

[andre@compaq-pc tmp]$ cmp foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm
foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm differ: byte 8081, line 26
[andre@compaq-pc tmp]$ cmp -i 8080 -l foo/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.i686.rpm
  1  22  16
  5  43   4
  6 270 120
  7 207 110
  8  54  66
[andre@compaq-pc tmp]$

[andre@compaq-pc tmp]$ cmp foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm
foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm differ: byte 8001, line 28
[andre@compaq-pc tmp]$ cmp -i 8000 -l foo/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm foo2/Packages/xorg-x11-drivers-7.4-2.fc15.x86_64.rpm
  1  22  16
  5  43   4
  6 270 120
  7 207 110
  8  54  66
[andre@compaq-pc tmp]$

This package uses the new compression in TC2. I was able to generate a working deltarpm between the old and new RPMs using {make,apply}deltarpm with the -r option, so have no idea why the copy in the rebuilt ISO is corrupted.

Comment 3 Michael Young 2011-02-15 17:55:04 UTC
The context is (the hexdump starts at 3498530554)
00002f0    3d65    6567    656e    6972    0063    7063    6f69    7800
00002f0   e   =   g   e   n   e   r   i   c  \0   c   p   i   o  \0   x
0000300    007a    0032    3878    5f36    3436    722d    6465    6168
0000300   z  \0   2  \0   x   8   6   _   6   4   -   r   e   d   h   a
0000310    2d74    696c    756e    2d78    6e67    0075    0000    0800
0000310   t   -   l   i   n   u   x   -   g   n   u  \0  \0  \0  \0  \b
0000320    0000    3f00    0000    0700    ffff    e0fd    0000    1000
0000320  \0  \0  \0   ?  \0  \0  \0  \a 377 377 375 340  \0  \0  \0 020
0000330    37fd    587a    005a    0a00    fbe1    a10c    0002    0121
0000330 375   7   z   X   Z  \0  \0  \n 341 373  \f 241 002  \0   ! 001
0000340    0012    0000    b823    2c87    00e0    007b    5d1d    1800
0000340 022  \0  \0  \0   # 270 207   , 340  \0   {  \0 035   ]  \0 030

where the original bytes are on the last line, so it looks to be near the start of an xz compression block.

Comment 4 Jonathan Dieter 2011-02-15 18:23:27 UTC
If you rpm2cpio the two different versions of the rpm, is the uncompressed data identical?

Comment 5 Jonathan Dieter 2011-02-15 18:36:11 UTC
Ok, scratch that last question.  Do you have a copy of the old xorg-x11-drivers rpm anywhere?  Can you give me a link?

Comment 6 Jonathan Dieter 2011-02-15 18:59:57 UTC
Forget it, I checked in koji and it was easy to see what the previous one was.  

I made a deltarpm between xorg-x11-drivers-7.4-1.fc14 and xorg-x11-drivers-7.4-2.fc15, then applied at using -r and got an md5 mismatch.

So it is a compression format change problem, but I don't know why it's showing up as xorg-x11-drivers-7.4-2.fc15 was built after we should have had updated versions of xz on our builders.

Maybe it would be a good idea to ask for another rebuild of xorg-x11-drivers and see what that looks like.

Comment 7 Michael Young 2011-02-15 20:31:11 UTC
I think the differences are the xz dictionary size parameter (1MiB verses 256KiB) for the LZMA2 filter and a 32-bit CRC. The package is small enough for this difference not to matter in terms of uncompressing.

Comment 8 Jonathan Dieter 2011-02-15 20:45:26 UTC
That sounds about right.  I was able to manually match the original xz output by using xz -4 rather than rpm's default level 3.  The question is, why was this rpm compressed using a different setting.  Deltarpm (and therefore deltaiso) *depends* on being able to recreate a byte-for-byte copy of the original compressed rpm, and that depends on compression being deterministic.

Comment 9 Michael Young 2011-02-16 00:30:55 UTC
Is deltarpm being too clever here? The file archive in xorg-x11-drivers are the same in both versions (both are actually empty), so could deltarpm be deciding to reuse the existing compressed cpio file rather than creating a new one. It would make sense for deltarpm to do this in most cases because it would avoid uncompressing and recompressing the contents only to end up with what you started with, and it would only be a problem if something changed in the compression algorithm.

Comment 10 Andre Robatino 2011-02-16 01:40:50 UTC
(In reply to comment #6)
> Forget it, I checked in koji and it was easy to see what the previous one was.  
> 
> I made a deltarpm between xorg-x11-drivers-7.4-1.fc14 and
> xorg-x11-drivers-7.4-2.fc15, then applied at using -r and got an md5 mismatch.

See the end of comment 2 - I previously copied xorg-x11-drivers-7.4-1.fc14 out of the loop-mounted TC1 ISO, xorg-x11-drivers-7.4-2.fc15 out of the loop-mounted TC2 ISO, then used {make,apply}deltarpm with the -r option and got a successful rebuild, with my F14 box modified to use the new compression as indicated in my original post. This worked on both i386 and x86_64. Were you using the old compression?

Comment 11 Jonathan Dieter 2011-02-16 09:20:09 UTC
Re: comment #10:
Aw crumbs, yeah, I'm doing this on an F14 box, so, yeah, of course it won't work.  /me smacks my forehead.

Re: comment #9:
You know, I think you might be right.  As far as I know, makedeltarpm *always* recreates the payload, but I noticed makedeltaiso saying that xorg-x11-drivers.i686 is "unchanged", so it may be reusing the compressed payload from xorg-x11-drivers-7.4-1.fc14.  Of course, given deltarpm's codebase, this will be a royal pain to fix.  Let me see what I can do.

Comment 13 Jonathan Dieter 2011-02-16 14:40:34 UTC
I've just checked the code, and, yes, makedeltaiso basically skips over rpms whose *uncompressed* payloads haven't changed.  Unfortunately, I'm hesitant to screw around with this code, as the easiest alternative is to have makedeltaiso delta *every* rpm on the DVD.  This would obviously take *much* longer and leave larger deltaisos to boot, and is therefore a nonstarter.

A better alternative would be to check whether the *compressed* data has changed after checking the uncompressed data, but this would either be time-consuming or a major code restructure (or possibly both).  Let me make contact with Michael S. upstream and see if he has any ideas.

Andre, how important is this bug?  We won't run into this problem with the next ISO (TC2->TC3), and the only reason we hit it this time is that we've changed the compression settings.

Comment 14 Andre Robatino 2011-02-16 15:19:58 UTC
I don't understand what's different between TC1->TC2 and TC2->RC1 (which is scheduled for tomorrow, unless it's delayed). Is it that this bug only happens due to this particular xorg-x11-drivers update? And does that mean that all deltas from F14 Final to later ISOs will be broken as well, even when all packages in the new ISO are using the new compression? (Although if that's the case, a fixed deltarpm wouldn't help much, since it would depend on RPM 4.9 making the F14 modification impractical. The version I used was an older one that still depended on RPM 4.8.)

Also, do you know what the situation is regarding future xz compression changes? Is it pretty much set in stone now, or still subject to change? Given the large potential savings from using delta compression on ISOs, it's a shame that all this can't be made to Just Work somehow (from the user's standpoint). Even going from (N-1) Final to N Final saves about 50% but if the compression changes every other year, that means one out of every 4 such deltas would be potentially unusable (since even without the current bug, there may be no practical way to modify an (N-1) system to use N's compression).

BTW, I tried pinging you several times on #fedora-qa.

Comment 15 Michael Young 2011-02-16 15:42:56 UTC
I wonder if you could have a more general "treat as new" list of packages where
if a package was on the list the delta iso program just ignored the old version
and included the full rpm, as if it were a new package rather than an update.
It would allow you to get around problems like this without needing to change
the code each time, and without extra processing in general.

It might be possible to do a one-off hack along these lines to generate a
TC1->TC2 delta iso that worked.

Comment 14:
TC2->RC1 should work unless any other similar "empty" packages have been updated between these two milestones. But you will have the same problem with F14->F15 (unless you do something sneaky and include a specially modified intermediate ISO that omits the xorg-x11-drivers package - your updates from F13 to F14 include several steps anyway so you could generate the first step, rebuild that iso without the xorg-x11-drivers package regenerate the diff to the modified iso, and then generate the diffs from the modified iso to the final iso as normal).

Comment 16 Jonathan Dieter 2011-02-16 17:59:11 UTC
I've sent off an email to upstream asking about a fix.

In the meantime, Michael, your idea in comment 15 should work for TC1 -> TC2 as well (assuming it works at all; I'm not sure if applydeltaiso checks checksums on the old ISO).

Comment 17 Michael Young 2011-02-16 19:36:17 UTC
(In reply to comment #15)
> It might be possible to do a one-off hack along these lines to generate a
> TC1->TC2 delta iso that worked.
I decided to try doing a one-off hack and the results are in koji at 
http://koji.fedoraproject.org/koji/taskinfo?taskID=2844891
I tried it on a pair of mini isos from the fc14 xorg-x11-drivers package and another rpm to the fc15 xorg-x11-drivers package and an rpm update, and it worked, so I expect it will generate a TC1->TC2 deltaiso that will work with the normal applydeltaiso program, though no promises. It might do FC14->TC2 or later as well though that depends on what else has changed.
It is a scratch build so it will be automatically deleted after about 2 weeks.

Comment 18 Michael Young 2011-02-16 22:54:20 UTC
(In reply to comment #16)
> In the meantime, Michael, your idea in comment 15 should work for TC1 -> TC2 as
> well (assuming it works at all; I'm not sure if applydeltaiso checks checksums
> on the old ISO).

applydeltaiso doesn't run a checksum on the source iso, but is depends closely on the contents so almost any difference in the source iso is likely to cause it to fail (the only thing I found that worked was changing a byte in the xorg-x11-drivers rpm which was ignored by the delta iso I generated with the hacked makedeltaiso).

What I meant in comment 15 was that you use two (or more) delta isos to get from the start iso to the end one, where the first intermediate iso is constructed to omit the xorg-x11-drivers rpm. If there are going to be intermediate isos anyway (as might be the case going from F14 to F15) it makes sense for this to be close to what would have been the first intermediate step.

Comment 19 Andre Robatino 2011-02-18 12:01:13 UTC
(In reply to comment #13)

> A better alternative would be to check whether the *compressed* data has
> changed after checking the uncompressed data, but this would either be
> time-consuming or a major code restructure (or possibly both).  Let me make
> contact with Michael S. upstream and see if he has any ideas.

I don't think makedeltaiso's speed is too important compared to being reliable in terms of generating working deltas. Applydeltaiso's speed is far more important. Anyway, I don't understand why it compares old and new rpms by comparing their uncompressed data. A direct comparison would be 100% reliable - wouldn't it also be faster as well? (I'm probably missing something so excuse me if this is a dumb question.)

Comment 20 Andre Robatino 2011-02-18 12:37:36 UTC
qemu-kvm-0.13.0-0.5.20100809git25fdf4a.fc15 is on TC2, contains no files, and is still using the old compression. so it appears this will happen again once.

Comment 21 Michael Young 2011-02-18 23:20:09 UTC
Created attachment 479629 [details]
Possible patch to compare compressed data

Here is another attempt at a fix, along the lines of comment #13 . It does what I would expect on my rather minimal test example. I have done a scratch build with the patch applied at http://koji.fedoraproject.org/koji/taskinfo?taskID=2850397 if you want to test it.

Comment 22 Andre Robatino 2011-02-19 09:36:49 UTC
Your patch works! I used it in a F15 VM (I can't update F14 to it since it depends on rpm-4.9) and makedeltaiso to generate the i386 TC1->TC2 DVD diso. The only difference in the screen output was that the xorg-x11-drivers line now says "creating delta" instead of "unchanged". The new diso works with applydeltaiso from the unpatched version. It's hard to tell whether makedeltaiso is slower due to having to run it in a VM.

I'll make and verify the x86_64 TC1->TC2 DVD diso as well and post both. Jonathan, if this patch looks good, could you push an update, and maybe a custom build that only depends on rpm-4.8 as well (so it can be used to update the F14 version)?

Comment 23 Jonathan Dieter 2011-02-19 10:53:14 UTC
Created attachment 479663 [details]
Attempt two on the patch to compare compressed data

The patch looks good to me.  The one thing I don't like is that we read in the new rpm both compressed and uncompressed even if it's byte-for-byte identical.  I'm assuming you were worried about namebuf[0] being set, but we set it to 254 anyway, so that shouldn't be a problem.

I've modified the patch so it doesn't uncompress (or reopen) anything if the compressed payloads are identical.  I've done a local test with a single-rpm iso and it works correctly, but, Andre, I'd appreciate it if you could test with TC1->TC2.

And, yes, users won't need an updated applydeltaiso.  Michael, many thanks for the patch.

Comment 24 Andre Robatino 2011-02-19 11:01:45 UTC
I attempted to build the 64-bit TC1->TC2 delta using Michael's patched version on my 64-bit F15 VM. The 32-bit delta build worked, but this time it used lots of CPU and I/O but didn't progress beyond

/tmp/foo/F15-Alpha.TC1/x86_64/Fedora-15-Alpha-x86_64-DVD.iso: 2920 rpms
Fedora-15-Alpha-x86_64-DVD.iso: 2925 rpms
reading old iso (omitting payloads)
size without payloads is 409533929 bytes
reading new iso (omitting payloads)
size without payloads is 411574724 bytes
creating iso diff

after almost an hour. I don't know if this would take extremely long due to my VM only having 1 GiB of RAM and being slower. Please do a build of the new patch and I'll try it. Thanks.

Comment 25 Jonathan Dieter 2011-02-19 11:20:14 UTC
http://koji.fedoraproject.org/koji/taskinfo?taskID=2851029

It's built against dist-f15

Comment 26 Michael Young 2011-02-19 12:23:25 UTC
(In reply to comment #23)
> I've modified the patch so it doesn't uncompress (or reopen) anything if the
> compressed payloads are identical.  I've done a local test with a single-rpm
> iso and it works correctly, but, Andre, I'd appreciate it if you could test
> with TC1->TC2.

Yes, I had assumed that payread might set more than the first character of namebuf without thinking to check.

(In reply to comment #22)
> It's hard to tell whether
> makedeltaiso is slower due to having to run it in a VM.

I would expect it to be marginally slower because in most cases it is doing extra reads and memory moves, but most of the time is probably used in uncompressing and compressing contents which is unchanged (except for any rpms like xorg-x11-drivers where we do save some processing but those will be rare). 

I have tested Jonathan's revised SRPM rebuilt for F14 (scratch build at 
http://koji.fedoraproject.org/koji/taskinfo?taskID=2851048 ) on my test case and it works for me too.

Comment 27 Andre Robatino 2011-02-19 12:58:57 UTC
I'm not able to use Jonathan's version either in a VM. I downgraded to the original F15 version and the same problem happens with that, so the VM must just be resource-starved. Jonathan: could you create a build that depends on rpm-4.8, but uses the new compression, so I could do the F14 modification as in http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ? It seems to be the only way to do the testing in a reasonable amount of time. Thanks.

Comment 28 Michael Young 2011-02-19 14:25:05 UTC
(In reply to comment #27)
> I'm not able to use Jonathan's version either in a VM. I downgraded to the
> original F15 version and the same problem happens with that, so the VM must
> just be resource-starved. Jonathan: could you create a build that depends on
> rpm-4.8, but uses the new compression, so I could do the F14 modification as in
> http://lists.fedoraproject.org/pipermail/test/2011-February/097009.html ? It
> seems to be the only way to do the testing in a reasonable amount of time.
> Thanks.

Well it isn't the only way. If your setup allows it you could mount the file systems from your VM on the parent box and run makedeltaiso under a chroot (it would see the files of the VM but with the memory of the parent box).

However I have also tried a rebuild of Jonathan's SRPM on F14 against the later xz version and uploaded the resulting rpms (except the debuginfo one) to http://myoung.fedorapeople.org/temp/ and I think it should do what you want.

Incidentally I don't think the dependency on rpm-4.8 actually matters for makedeltaiso (though it might for applydeltaiso) so the patched F15 executable may work.

Comment 29 Andre Robatino 2011-02-19 16:31:32 UTC
Jonathan explained to me on IRC how to build his source RPM on my F14 box, which I did, then installed it. Then I ran {make,apply}deltaiso for TC1->TC2 x86_64. The result was very strange: it claimed the ISO was corrupt, but in fact it was good. It turned out that makedeltaiso was storing the wrong md5 sum:

iso diff done, final compression: 17.1%
iso md5sum is: 971cc87e295b172010c3a98be0792c16

when in fact the correct md5 sum for TC2 x86_64 is

ac0ea9a27177d1ef44ab4c16524b1bbf

which is what the unpatched makedeltaiso reports.

Comment 30 Jonathan Dieter 2011-02-19 16:49:36 UTC
Ok, I see the problem.  Most of the MD5 checksum comes from reading the uncompressed data, so, Michael, you were right, we do need to read in the uncompressed data no matter what.  Ugh.

Comment 31 Andre Robatino 2011-02-19 20:01:45 UTC
Compiling the src rpm from http://koji.fedoraproject.org/koji/taskinfo?taskID=2850397 on my F14 box with the new xz compression and using the resulting binary rpms for both makedeltaiso and applydeltaiso, it works properly for TC1->TC2, both i386 and x86_64. Running both on bare metal means I can do a direct speed comparison of unpatched vs. patched. If anything, the patched version is a little faster.

[andre@compaq-pc ~]$ diff makedeltaiso_TC2_i386.txt makedeltaiso_TC2_i386_patched.txt
2840c2840
< xorg-x11-drivers.i686: unchanged...13.6%
---
> xorg-x11-drivers.i686 (xz.2): creating delta...13.6%
2941,2943c2941,2943
< real	31m11.871s
< user	25m14.049s
< sys	1m49.963s
---
> real	29m29.843s
> user	24m12.433s
> sys	2m14.964s
[andre@compaq-pc ~]$ diff makedeltaiso_TC2_x86_64.txt makedeltaiso_TC2_x86_64_patched.txt
2838c2838
< xorg-x11-drivers.x86_64: unchanged...16.7%
---
> xorg-x11-drivers.x86_64 (xz.2): creating delta...16.7%
2937,2939c2937,2939
< real	33m30.262s
< user	28m25.854s
< sys	2m22.838s
---
> real	32m48.911s
> user	27m24.683s
> sys	1m27.068s
[andre@compaq-pc ~]$

Comment 32 Andre Robatino 2011-02-23 10:13:42 UTC
The qemu-kvm package was not updated yet in RC1, and as expected the TC2->RC1 deltas generated by the unpatched and patched versions of deltarpm are identical and both work. (See comment 20.) I posted both them and the working TC1->TC2 deltas on my download page ( https://fedoraproject.org/wiki/User:Robatino/Downloads ).

Comment 33 Andre Robatino 2011-02-23 10:39:10 UTC
Jonathan: could you push fixed versions of deltarpm before Michael's scratch build in comment 21 from 2/18 expires (in case someone else needs the patch - I've already saved a copy of the src rpm)? Thanks.

Comment 34 Fedora Update System 2011-02-23 11:01:40 UTC
deltarpm-3.6-0.6.20110223git.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc14

Comment 35 Fedora Update System 2011-02-23 11:03:01 UTC
deltarpm-3.6-0.6.20110223git.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc15

Comment 36 Fedora Update System 2011-02-23 20:28:25 UTC
deltarpm-3.6-0.6.20110223git.fc15 has been pushed to the Fedora 15 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update deltarpm'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/deltarpm-3.6-0.6.20110223git.fc15

Comment 37 Fedora Update System 2011-02-26 03:59:42 UTC
deltarpm-3.6-0.6.20110223git.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 38 Fedora Update System 2011-03-03 03:36:19 UTC
deltarpm-3.6-0.6.20110223git.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.