Bug 2229654

Summary:	rsync - buffer overflow detected
Product:	Red Hat Enterprise Linux 8	Reporter:	dpawlik <dpawlik>
Component:	rsync	Assignee:	Michal Ruprich <mruprich>
Status:	ASSIGNED ---	QA Contact:	rhel-cs-infra-services-qe <rhel-cs-infra-services-qe>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.7	CC:	debarshir, fboucher
Target Milestone:	rc	Flags:	mruprich: needinfo? (dpawlik)
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description dpawlik 2023-08-07 08:43:21 UTC

Description of problem:

On executing rsync command,I receive an error:

*** buffer overflow detected ***: terminated
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3]

on executing command: 

/usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group --rsh=/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null /tmp/toolbox zuul-worker:~/src/


Version-Release number of selected component (if applicable):

On host, where the rsync is executed: rsync-3.1.3-19.el8_7.1.x86_64
Remote host (Fedora Rawhide): rsync-3.2.7-4.fc39.x86_64

Steps to Reproduce:

* By using shell:

1. git clone https://github.com/containers/toolbox /tmp/toolbox
2. /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group --rsh=/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null /tmp/toolbox zuul-worker:~/src/

* By using Ansible (2.9.27):

1. git clone https://github.com/containers/toolbox /tmp/toolbox
2. Create playbook + inventory:

cat << EOF > test.yaml
---
- name: test
  hosts: test.dev
  gather_facts: false
  tasks:
  - name: Synchronize src repos to workspace directory.
    synchronize:
      delete: true
      dest: "~/src/"
      recursive: true
      src: "/tmp/toolbox"
      owner: no
      group: no
EOF

cat << EOF > inventory.yaml
---
all:
  hosts:
    test.dev:
      ansible_port: 22
      ansible_host: <Fedora Rawhide ip address>
      ansible_user: zuul-worker
EOF

3. ansible-playbook -i invenory.yaml test.yaml 


Actual results:

rsync: connection unexpectedly closed (15 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]

Expected results:

Everything is synced.

Comment 1 Fabien Boucher 2023-08-14 08:12:20 UTC

The affected product has been set to rhel8 however the issue is likely to have been introduced with the last versions of rsync into Fedora rawhide.

Comment 2 Michal Ruprich 2023-08-15 12:53:44 UTC

Hi,

so you say that this started with version 3.2.7 but this version is in Fedora for over 10 months, was this really the first time this started to happen? This is definitely something in the new version, it crashes even between 3.2.7 and 3.2.7 on both sides.

Can you just try to compare the directory structures on both sides even after the crash? Seems to me that everything is actually transferred but it crashes after the transfer.

Regards,
Michal

Comment 3 Michal Ruprich 2023-08-16 10:05:13 UTC

Correction, everything is not sent, I was probably looking at wrong output. I would like to ask you why are you using the -F option? Do you need it to filter something? Looking at the definition:

-F     The -F option is a shorthand for adding two --filter rules to your command.  The first time it is used is a shorthand for this rule:

                  ‐‐filter=’dir‐merge /.rsync‐filter’

       This  tells  rsync  to look for per‐directory .rsync‐filter files that have been sprinkled through the hierarchy and use their rules to filter the files in the transfer.

I don't see any such files in the source location and without -F there is no crash. There is definitely a bug here but this might be a suggestion for temporary workaround.

Comment 4 Fabien Boucher 2023-08-16 10:08:53 UTC

Hi,

So it seems that even if the rsync is the same version then the base system introduce a behavior change when using the "--delete-after" option on rawhide.

Here is the log of a new investigation:

Sender node: rsync-3.1.2-12.el7_9.x86_64
Receiver node: rsync-3.2.7-4.fc39.x86_64

Running:

git clone https://src.fedoraproject.org/rpms/python-gear
/usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group python-gear zuul-worker.83.xxx:/tmp/test-1
*** buffer overflow detected ***: terminated
^CKilled by signal 2.
rsync error: unexplained error (code 255) at rsync.c(638) [sender=3.1.2]

On both side the output of "find python-gear | wc -l" is similar (48) then it seems the transfer was complete.

Also note that:

/usr/bin/rsync -v --delay-updates -F --compress --archive --no-owner --no-group python-gear zuul-worker.83.xxx:/tmp/test-4
Running the same command but without the "--delete-after" option the rsync command complete with success.


Running the same rsync command but the receiver in now (same sender):
$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)
$ rpm -qa | grep rsync
rsync-3.2.7-2.fc38.x86_64

/usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group python-gear zuul-worker.83.yyy:/tmp/test-1
The command run with success.

Comment 5 Fabien Boucher 2023-08-16 10:15:35 UTC

Yes removing "-F" or "--delete-after" avoid the overflow issue.

All this options are set in the test command because those options are set by the Ansible synchronize module as used by our CI: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace/tasks/main.yaml

Comment 6 Michal Ruprich 2023-08-16 10:22:23 UTC

This quite interesting. There is absolutely no difference in code between rsync-3.2.7-2 and rsync-3.2.7-4 which makes this even more interesting.

Comment 7 Michal Ruprich 2023-08-17 12:36:18 UTC

Most likely the same bug already filed in Upstream - https://github.com/WayneD/rsync/issues/511