Red Hat Bugzilla – Bug 495310
rsync contains forked copy of zlib
Last modified: 2013-08-07 07:11:41 EDT
Description of problem:
rsync is carrying its own copy of zlib. This is not allowed for reasons noted here:
and in the preceding section on static libraries.
Additionally, other programs are appearing which wish to use that code. zsync makes its own copy of the rsync library and librsync has discussed embedding but appears to have gone with a less efficient method that uses the system zlib for now.
The rsync zlib is a forked copy. The zlib/README.rsync file has the following details:
Specific changes that have been made to zlib for rsync include:
- add Z_INSERT_ONLY to allow for efficient history updating without
actually emitting any data. This is used to compress the matched
blocks that don't cross the wire, which gives better compression
ratios on the literal data.
- fixed a number of minor compilation issues. (redefinition of MAX and
other such trivial things)
- include rsync.h to ensure that we get a consistent set of includes
for all C code in rsync and to take advantage of autoconf
The first is the divergence that causes other programs wanting to speak the rsync protocol to wish to use this version of the library.
1) Get these changes into upstream zlib. The README.rsync has this note:
The rsync maintainers hope to fix this problem in the future by either
merging our changes into the upstream version, or backing them out of
rsync in a way that preserves wire compatibility. But in the meantime
this version must be maintained in parallel.
but this seems to never have happened. The diff between vanilla zlib and rsync zlib as of 2005:
2) Get upstream rsync to drop its copy and use system zlib. This is also something upstream has discussed but has either not found a way to do compatibly or hasn't had the will to achieve.
3) Get the forked zlib into its own library. If we had a library like libz_rsync we'd have the following advantages:
* Only one copy on the system instead of one for every application.
* Security and other fixes would flow from upstream to all the applications via normal dynamic library linking instead of a sequence of upstream's having to notice the changes.
* If we're lucky enough to get upstream rsync maintaing this, perhaps they'll rethink how much work it is vs getting their changes merged to zlib :-)
We'd still be depending on the libz_rsync maintainers tracking zlib in a timely fashion, though.
I honestly think this is something you have to pursue upstream.
I don't think it would be a good idea to carry patches to do something like that as a private Fedora modification.
<nod> I think it belongs upstream as well. The question is who is responsible for carrying that upstream? It's usually the package maintainer who is responsible. However, this usually comes up at review time and it looks like rsync wasn't reviewed by someone who checked this.
BTW, this has come up because it's blocking the zsync review:
I've mentioned that they should try and carry this work to rsync upstream here:
Perhaps this idea is good, perhaps this is totally bullshit. If it is totally bullshit please ignore it.
There are so many forked zlib and every upstream has his own changes and this would be impossible to include all changes.
We should make extra pacakges for every zlib with the project that is forked for as suffix or prefix. like rsync-zlib or zlib-rsync with the dependency of this package. Just an idea, please don't stone me to death for this.
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.
More information and reason for this action is here:
notting reported at the FESCo meeting a couple months ago that you thought we could fix this without too much difficulty downstream. What's going on with this?
I don't remember talking with Notting about this.
And I am not comfortable forking off a library downstream and upstream seem uninterested in dealing with this stuff.
If someone wants to do the job and the patching be my guest though.
My preferred course of action though, would be to get whatever modification necessary included in upstream zlib and not in a distribution specific fork, and then ask upstream to use the available library.
Created attachment 365316 [details]
patch for building zlib as a shared lib
Created attachment 365317 [details]
patch to create subpackages (lib + devel) with the forked copy of zlib
Simo, because we should move somewhere I prepared a patch that will build the internal copy of zlib as a separate shared library and link rsync against it. The other patch adds 2 subpackages to rsync to distribute the library and the development files.
The whole work with creating and installing the shared library is put into the Makefile patch and the spec file patch is minimalized, but it could be also done the other way with minimal Makefile patch and more work in the spec file.
One note to original spec file - setting the CFLAGS when "make" is called is here a bad thing, because the right CFLAGS are already set via %configure directly into the generated Makefile.
Dan, that's not really the expectation. Rsync should link against the system
zlib or an official fork that can be used by rsync and zsync as well. Current
result is not really the solution from my point of view...
Robert, but my opinion is this is a distribution of a fork that can be used by zsync (and eventually others) and building both rsync and the library from one source archive is desirable. Or do you prefer to have a separate source archive and package for the forked zlib? Accepting the fact that using a system zlib is not possible in short or mid term ...
Well, is your work a rsync-only thing or can it be used by zsync as well? If
it's a rsync-only thing, we unfortunately IMHO have nothing gained. Because if
I now need a zsync-zlib/zsync-zlib-devel as well, we wouldn't have needed a
shared library being anyway the same rsync fork of zlib.
In zsync you will need to have rsync-zlib-devel as BuildRequires: and patch the buildsystem a bit to link with librsync-zlib and use headers from /usr/include/rsync-zlib. I didn't check the sources of zsync, but there shouldn't be an issue. Or do I miss something?
Looks like I missed something :-( While the rsync and zsync applications are wire-compatible, the rsync-zlib and zsync-zlib are not API compatible, OMG ...
The diffs against upstream versions can be seen at http://fedora.danny.cz/rsync Maybe the changes could be merged into one "super-zlib" ...
gcc -std=gnu99 -o librsync-zlib.so.1 -shared -Wl,--soname,librsync-zlib.so.1 zlib/deflate.o zlib/inffast.o zlib/inflate.o zlib/inftrees.o zlib/trees.o zlib/z
util.o zlib/adler32.o zlib/compress.o zlib/crc32.o
/usr/bin/ld: zlib/deflate.o: relocation R_X86_64_32S against `_length_code' can not be used when making a shared object; recompile with -fPIC
zlib/deflate.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [librsync-zlib.so.1] Error 1
At first, your patch for rsync is unfortunately not usable at x86_64 right now.
(In reply to comment #17)
> At first, your patch for rsync is unfortunately not usable at x86_64 right now.
Robert, you must remove the CFLAGS override from the make call in %build section of the spec. It not required even in normal rsync builds and creates the link error you see.
FWIW, I entered this upstream: https://bugzilla.samba.org/show_bug.cgi?id=6916
Jan Zeleny is the new maintainer of rsync.
Jan, please take care of this bug and finally solve it.
Matt, you opened a bug, but it seems that nothing happened. Please ask upstream about the current state. Perhaps they have talked about it and the result of this talk is not noticed in the bug.
the most important of all questions is:
So what is Plan B if upstream won't fix this bug?
patching rsync to build against syslib will imho fail, because you will lose compatibility to rsync itself (for using compression compress). we can't remove it, because rsync is a standard package for each distribution.
on the other hand. our guidelines are 100% correct in a philosophical way, but sometimes technical inpracticable. It is not our duty to educate upstream projects. We can only give hints, criticize parts of the code and give suggestions for improvement, but we can't set framework conditions for upstream projects. Imho we have to improve our guidelines.
Another thing is that our FESCO decided arbitrary, because they allow that some forked libs can stay (happened in wordpress bundled libs bug) and other forked libs must be replaced (happened in the rsync and zsync bugs). These decisions are not reasonable for me.
(In reply to comment #20)
> Matt, you opened a bug, but it seems that nothing happened. Please ask upstream
> about the current state.
Sorry, I didn't intend to accept responsibility for pushing this upstream.
(In reply to comment #21)
> Sorry, I didn't intend to accept responsibility for pushing this upstream.
If you won't accept the responsibility why you opened this bug in upstreams bugzilla? just open a bug and look what will happened won't help to solve the problem.
(In reply to comment #22)
> If you won't accept the responsibility why you opened this bug in upstreams
Because I thought the complaint was legitimate and it would be helpful to have it recorded in the upstream tracker, nothing more. My personal interest in working to solve the problem is minimal at this time.
(In reply to comment #20)
> Jan Zeleny is the new maintainer of rsync.
> Jan, please take care of this bug and finally solve it.
> Matt, you opened a bug, but it seems that nothing happened. Please ask upstream
> about the current state. Perhaps they have talked about it and the result of
> this talk is not noticed in the bug.
It seemed that we might be able to achieve some progress, but now I think zlib and rsync upstreams didn't actually start to communicate to solve this issue. And to solve it, communication is exactly what is needed.
> the most important of all questions is:
> So what is Plan B if upstream won't fix this bug?
I'm not sure. I think some discussion is really the most important part and the only acceptable solution. We cannot create our version of either component to solve this in Fedora before we know what is right for both packages. And we cannot decide what's best for upstreams.
CC'ing Ivana, she is maintainer of zlib, so she might be interested in this bug as well
Note: I searched through the zlib mailing list archives and did not see that the rsync changes were ever submitted to zlib. (You have to apply to the mailing list, then email Mark Adler to get approved. Instructions on the mailman info page: http://www.zlib.net/mailman/listinfo/zlib-devel_madler.net)
Someone could have submitted it directly to madler or Jean-Loupe but that's not really the same thing.
I think someone needs to pursue rsync upstream to get and understand the patch to the bundled zlib and then submit that to the zlib upstream.
If that fails (or if it looks like a long process) we can also fork off the rsync bundled zlib into it's own library and have rsync and zsync use it in common. (note that rsync and zsync modify the zlib library in different ways even though both are supposed to have the same output :-( If we fork, we should get buy in from the zsync upstream and the Debian/other distros. At least Debian has noted that this is a problem before so we probably wouldn't need to maintain this alone.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Vojtěch, you are the owner of this packages for more than two months. what have you done so far to solve this violation of our guidelines?
Simon, I was investigating on this mostly previous month, after that I had some other urgent work to do.
I had a talk with Jan Zeleny about the situation in rsync/zlib upstream and of course I read all relevant mailing lists.
I have also subscribed myself to zlib-devel mailing list today and I am waiting for Mark Adler's approval.
I am also going to talk with Ivana Varekova, fedora zlib maintainer, about our upcoming co-operation.
1) We should probably measure compression efficiency of rsync-zlib Z_INSERT_ONLY to pursue zlib upstream to merge it. This would also require implementing the receiver-side part in zlib inflate functions. This is now hacked in rsync source code only.
2) If the first step fails, we should pursue rsync upstream to get rid of this optimization and start using system's zlib. They were actually talking about this possibility earlier in the above mentioned mailing list.
Rsync is now missing lot of fixes that have been made between zlib 1.2.3 and current 1.2.5. See http://www.zlib.net/ChangeLog.txt
I will inform you about any progress asap.
Hi Vojtech, what's the status?
Hi Thom, please accept my apologies for my non-responsiveness.
Few weeks ago, I had another speech with Jan. He explained me the previous progress in communication between rsync and zlib upstreams - as far as what he could remember. Unfortunately he no longer has any copy of the old emails.
Finally, I was also talking with Ivana, the Fedora zlib maintainer. She explained to me the situation between zsync and zlib upstreams. She has also showed me some details from old communication between Colin Phipps (zsync) and Mark Adler (zlib).
When talking about the situation between rsync and zlib, she just referred to the mailing list mentioned above in the bz description.
To sum this up, it seems that no one from Fedora folks, who was involved in this issue before, knows any further details about any communication between upstreams themselves.
I am going to initiate a new communication between those and I will also ask the status from the past (if there is any). But before this step I want to come with some ideas / proposed solutions. That's why I spent some time on understanding the rsync rolling checksums algorithm, Z_INSERT_ONLY optimization in the zlib deflate.c, the rsync "hack" in token.c and of course, on lots of debugging.
I will focus on this issue more in depth and I will let you know about further steps.
Update on this issue:
I have initiated a fresh conversation between Wayne Davison (rsync) and Mark Adler (zlib) off list /with few others CC'ed/. They are now discussing possible changes in zlib (deflate/inflate add dictionary context patches) to make rsync functionality inclusion possible.
I will let you know about further progress.
We are now waiting for zlib upstream response to proposed patches..
Any word from zlib upstream?
> Subject: Re: Extending zlib to make unsent/shared data (ala rsync --compress) official?
> On Wed, May 18, 2011 at 6:41 AM, Vojtech Vitek <vvitek AT redhat DOT com>
> > was there any off list update on the topic, please?
> Sadly, I haven't heard anything back yet.
I've asked wayne again on July 11 - sadly with no response back yet..
I'm going to ping both upstreams again right now..
Still waiting for zlib developers..
Unfortunately, they didn't comment the proposed patches yet, nor they responded to my email from Aug 8 2011.
zsync is in the rpmforge repo just go to http://pkgs.repoforge.org/rpmforge-release/ download the latest el6 for your architecture then install it and then you can install zsync via terminal command as root yum install zsync the list of packages they have is http://pkgs.repoforge.org/ and zsync is at http://pkgs.repoforge.org/zsync/ i can confirm it works for 64 bit fedora 16 beta rc4 fully updated
The fact that rpmforge offers zsync wouldn't solve the problem that rsync and zlib ship their own and modified zlib. This isn't compatible with the fedora packaging guidelines...
Hi Vojtech, please try pinging the zlib developers again.
Zlib upstream pinged on Oct 03 2011.
Zlib upstream responded on Oct 5 2011.
Now waiting for rsync upstream to answer several questions..
Rsync upstream pinged on Oct 29 2011.
The conversation have just rapidly moved on.. seems like /hopefully/ we could have some results soon.
== zlib ==
Zlib now permits use of inflateSetDictionary() and deflateSetDictionary() in the middle of a stream to insert dictionary data. There is also deflateResetKeep() function to retain history data across deflates. (There was already an inflateResetKeep().
Attached is a new version of reconcile.c that makes use of these capabilities, so that it does not need to maintain its own sliding history when compressing and decompressing. reconcile.c does not use flush operations to end a deflate block, instead using the built-in self-termination of deflate streams. So full flushes and the insertion and deletion of empty stored block bytes is not needed to bring the compressed data to a byte boundary.
Rsync upstream has been working on --with-included-zlib=no configure option in their development branch. I'll test the changes and provide feedback asap.
Whats the current status? What happened the last 2.5 months? Now, that yet
another Red Hat employee gave up rsync maintainance within these 3 years...
I will discuss the current state with Vojtech and inform you.
I am waiting for upstream response.
rsync-3.0.9-4.fc18 has been submitted as an update for Fedora 18.
rsync-3.0.9-3.fc17 has been submitted as an update for Fedora 17.
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rsync-3.0.9-4.fc18'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
Michal, how was this resolved, what about compatibility?
Specifically I would like to know:
1. Is the new rsync fully compatible with older versions of rsync?
2. Zsync package (see bug 490140) used to bundle rsync-specific zlib library to stay compatible with it. Can it now drop the dependency and still be compatible with older rsync versions, or does rsync-3.0.9-4 (or newer) needs to be deployed to the server for unbundled zsync to work?
I have not found any incompatibility during my internal testing(just basic tests). I admit that this update was probably too hurried and further reconsideration will be needed. Therefore, I am stopping the update process for now.
The current rawhide version(3.1.0-1pre1) of Rsync has been built with configure option --included-zlib=no and should be fully compatible with previous rsync versions.