Bug 2043753
| Summary: | [RFE] Improve defaults for sparse file buffering. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Mason Loring Bliss <mbliss> |
| Component: | rsync | Assignee: | Michal Ruprich <mruprich> |
| Status: | CLOSED ERRATA | QA Contact: | Ondrej Mejzlik <omejzlik> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.5 | CC: | amepatil, apevec, egolov, ekohlvan, farrotin, fkrska, fsumsal, h1k6zn2m, lance, leonfauster, mandreou, omejzlik, ondrejj, rhn, rpm |
| Target Milestone: | rc | Keywords: | FutureFeature, Triaged |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | rsync-3.1.3-17.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-11-08 10:55:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2089955 | ||
| Deadline: | 2022-08-08 | ||
Sorry for the delay here, I was hoping I would be able to reproduce this but after many attempts I don't see that much of a difference between sparse and non-sparse. Nevermind, the patch for setting up custom sparse-block size is in the upstream repo, only it is in the sort-of-unsupported rscyn-patches. I think it is ok to introduce this as a downstream change since this does not change any default usage and only for users who want to exploit the functionality could this feature make a difference. I have a suspicion this causes a regression when using rsync with this patch applied against an rsync daemon that doesn't support the --sparse-block option, see https://pagure.io/centos-infra/issue/872. This change breaks compatibility with older version of rsync. Running rsync from CentOS 8 stream to CentOS 7 is not possible without special parameters. Versions of components: rsync-3.1.3-16.el8.x86_64 rsync-3.1.2-10.el7.x86_64 What happens when trying to rsync: [backuppc@backup ~]$ rsync -vvvv root.sk:/etc/hosts /tmp/test/ cmd=<NULL> machine=devel.xxxx.sk user=root path=/etc/hosts cmd[0]=ssh cmd[1]=-l cmd[2]=root cmd[3]=devel.xxxx.sk cmd[4]=rsync cmd[5]=--server cmd[6]=--sender cmd[7]=-vvvve.LsfxC cmd[8]=--sparse-block=1024 cmd[9]=. cmd[10]=/etc/hosts opening connection using: ssh -l root devel.xxxx.sk rsync --server --sender -vvvve.LsfxC --sparse-block=1024 . /etc/hosts (11 args) msg checking charset: UTF-8 rsync: on remote machine: --sparse-block=1024: unknown option rsync error: syntax or usage error (code 1) at main.c(1568) [server=3.1.2] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] [Receiver] _exit_cleanup(code=12, file=io.c, line=226): entered rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.3] [Receiver] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(12) It is possible to workaround this problem by adding --sparse-block=0 to rsync parameters, but this is too complicated and breaks some backup programs. Please, revert this change. Also doesn't help to update rsync on CentOS 7, because there are other distributions, which can use rsync without this parameter. I did not notice that this option is sent to the server every time. I am now working on a fix. We (CentOS infra team) also confirm that we're suffering from that issue too, and that impacts the proper mirror workflow for CentOS Stream 8 and 9. Can that package be removed from CentOS Stream 8 for the time being ? +1 also affected I can confirm this behaviour. CentOS Stream 8, rsync local 3.1.3-16 Without having set the opsion --sparse-block, I get this nonetheless: rsync: on remote machine: --sparse-block=1024: unknown option rsync error: requested action not supported (code 4) at clientserver.c(1556) [Receiver=3.0.9] rsync: read error: Connection reset by peer (104) rsync error: error in socket IO (code 10) at io.c(785) [sender=3.1.3] adding info as i don't see it on the BZ already this is hitting tripleo image build jobs on centos8 - upstream bug at https://bugs.launchpad.net/tripleo/+bug/1984035 workaround for now is to downgrade rsync to 3.1.3-14 https://review.rdoproject.org/r/c/config/+/44471 Hi, I am putting this here as a public comment - we know about the issue and the fix is already out - rsync-3.1.3-17.el8. You can either downgrade to rsync-3.1.3-15.el8 where the feature is not present or wait till the -17 version gets to the CentOS. build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2114622 I know this affects a lot of systems, no need to add more and more systems that are affected here as comments. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: rsync security and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7793 |
1. Proposed title of this feature request Improve defaults for sparse file handling. 2. Who is the customer behind the request? Account: Allianz Technology SE / 888955 TAM customer: yes (Stefan Meyer / EMEA) CSM customer: (not indicated) Strategic: (yes) 3. What is the nature and description of the request? rsync uses inadequate defaults for sparse file buffering, which can be dramatically improved. 4. Why does the customer need this? (List the business requirements here) Large file transfers are unacceptably slow without adjusting buffer handling unfortunately, this requires a recompile. 5. How would the customer like to achieve this? (List the functional requirements here) Update buffer sizing. This was explored, for example, in this third- party bug: https://bugzilla.samba.org/show_bug.cgi?id=8512 With this: --------------------------------------------------------------------------- % diff -u rsync.h-org rsync.h --- rsync.h-org 2014-04-13 19:36:59.000000000 +0200 +++ rsync.h 2014-09-08 16:20:41.427973852 +0200 @@ -131,11 +131,11 @@ #define RSYNC_PORT 873 -#define SPARSE_WRITE_SIZE (1024) -#define WRITE_SIZE (32*1024) -#define CHUNK_SIZE (32*1024) +#define SPARSE_WRITE_SIZE (128*1024) +#define WRITE_SIZE (128*1024) +#define CHUNK_SIZE (128*1024) #define MAX_MAP_SIZE (256*1024) -#define IO_BUFFER_SIZE (32*1024) +#define IO_BUFFER_SIZE (128*1024) #define MAX_BLOCK_SIZE ((int32)1 << 17) --------------------------------------------------------------------------- With that, the user notes a factor of fifty increase. The customer tested with only SPARSE_WRITE_SIZE bumped, but not bumped as far: +#define SPARSE_WRITE_SIZE (32*1024) ...and noted significant performance impact: --------------------------------------------------------------------------- without "-S" sent 9,324,792,039 bytes received 42 bytes 90,973,581.28 bytes/sec unpatched "-S" sent 9,324,792,039 bytes received 42 bytes 8,190,419.04 bytes/sec patched with "-S": sent 9,324,792,039 bytes received 42 bytes 34,345,458.86 bytes/sec --------------------------------------------------------------------------- 6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. Customer can verify performance for large sparse transfers. 7. Is there already an existing RFE upstream or in Red Hat Bugzilla? Not that I've found. 8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL8, RHEL9)? Not specified. 9. Is the sales team involved in this request and do they have any additional input? Not as yet. 10. List any affected packages or components. rsync 11. Would the customer be able to assist in testing this functionality if implemented? Yes.