Bug 1212546

Summary: rsync-out fails to synchronize filesystem with remote filesystem
Product: [Community] Virtualization Tools Reporter: Hu Zhang <huzhan>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: NEW --- QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedCC: ksrot, leiwang, luhliari, mruprich, mzeleny, ovasik, ptoscano, rjones, ssorce, virt-maint, wshi, yoguo
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1432899    
Bug Blocks: 1224209    

Description Hu Zhang 2015-04-16 15:33:56 UTC
Description of problem:
rsync-out fails to synchronize filesystem with remote filesystem

Version-Release number of selected component (if applicable):
kernel-4.0.0-0.rc1.git0.1.fc22.x86_64
libguestfs-1.29.33-1.fc22.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Prepare the file on the host
# rm -fr /tmp/hostsrc
# rm -fr /tmp/hostdest
# mkdir /tmp/hostsrc
# mkdir /tmp/hostdest
# chmod 777 /tmp/hostdest
# touch /tmp/hostsrc/rsyncin_test.txt

2. Configure the rsyncd service.
# echo "port = 33335
pid file = rsyncd.pid
[src]
path = /tmp/hostsrc
comment = source
use chroot = false
read only = true
[dest]
path = /tmp/hostdest
comment = destination
use chroot = false
read only = false
" > /tmp/rsyncd.conf

# rsync --daemon --config=/tmp/rsyncd.conf

3. guestfish --network -a $GSTEST_image_path
><fs> run
><fs> mount-options noatime $mount_device /
><fs> rm-rf /dest
><fs> rm-rf /src
><fs> mkdir /src
><fs> mkdir /dest
><fs> touch /dest/rsyncout_test.txt
><fs> rsync-out /dest rsync://root@$ipaddr_of_host:33335/dest/ archive:true

Actual results:
libguestfs: error: rsync_out: '/dest' to 'rsync://root.10.241:33335/dest/': rsync: chgrp "dest" (in dest) failed: Operation not permitted (1)
rsync: chgrp "dest/.rsyncout_test.txt.AUEUIM" (in dest) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

Expected results:
rsync-out can synchronize filesystem with remote filesystem successfully

Additional info:

Comment 1 Richard W.M. Jones 2015-04-16 15:44:21 UTC
Is libguestfs-rsync installed (from usual Fedora yum repos)?

Comment 2 Richard W.M. Jones 2015-04-16 15:49:19 UTC
I tried the above commands, and it worked OK for me with:

rsync-3.1.1-3.fc21.x86_64
libguestfs-1.29.34-1.fc23.x86_64
libguestfs-rsync-1.29.34-1.fc23.x86_64

However I didn't try this on Fedora 22 yet (maybe rsync has
been broken).

Also I didn't try running rsync as root.

Comment 3 Richard W.M. Jones 2015-04-16 15:56:38 UTC
With rsync running as root, and the hostdest directory also owned by
root, I get a slightly different error:

><fs> rsync-out /dest rsync://192.168.0.175:2999/dest archive:true
libguestfs: error: rsync_out: '/dest' to 'rsync://192.168.0.175:2999/dest': rsync: recv_generator: mkdir "dest" (in dest) failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

Not completely convinced this isn't something to do with rsync
itself.

Comment 4 Hu Zhang 2015-05-06 02:21:48 UTC
(In reply to Richard W.M. Jones from comment #1)
> Is libguestfs-rsync installed (from usual Fedora yum repos)?

Yes, the version is libguestfs-rsync-1.29.33-1.fc22.x86_64

Comment 5 Hu Zhang 2015-05-06 03:20:17 UTC
rsync-3.1.1-6.fc22.x86_64
libguestfs-rsync-1.29.33-1.fc22.x86_64
libguestfs-1.29.33-1.fc22.x86_64

On fedora22.
1. Try with "test", a non-system user. The "hostdest" directory has the permission 775.
$ pwd
/home/test
$ ll
drwxrwxr-x. 2 test test          6 May  6 10:42 hostdest
drwxrwxr-x. 2 test test          6 May  6 10:13 hostsrc

### Now the "hostdest" does not have the o+w permission. 

Try with this, I will have a same result as yours, and it's same on rhel7.1.
><fs> rsync-out /dest rsync://test.11.53:33335/dest/ archive:true
libguestfs: error: rsync_out: '/dest' to 'rsync://test.11.53:33335/dest/': rsync: recv_generator: mkdir "dest" (in dest) failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

2. Try with "test", a non-system user. The "hostdest" directory has the permission 777.
$ pwd
/home/test
$ ll
drwxrwxrwx. 2 test test          6 May  6 10:42 hostdest
drwxrwxr-x. 2 test test          6 May  6 10:13 hostsrc

Try with this, I will get the result like:
><fs> rsync-out /dest rsync://test.11.53:33335/dest/ archive:true
libguestfs: error: rsync_out: '/dest' to 'rsync://test.11.53:33335/dest/': rsync: chgrp "dest" (in dest) failed: Operation not permitted (1)
rsync: chgrp "dest/.rsyncout_test.txt.Z2mwvR" (in dest) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

$ ll hostdest/
total 0
drwx------. 2 nobody nobody 30 May  6 10:10 dest

$ ll hostdest/dest/
ls: cannot open directory hostdest/dest/: Permission denied
### You can see the permission of "dest".

$ su
# ll hostdest/dest/
total 0
-rw-------. 1 nobody nobody 0 May  6 11:01 rsyncout_test.txt

### So, the file is copied out successfully, but the permission is not reasonable.

3. Let try the same scenario as step2 on rhel7.1.
$ pwd
/home/test
$ ll
drwxrwxrwx. 2 test test          6 May  6 10:42 hostdest
drwxrwxr-x. 2 test test          6 May  6 10:13 hostsrc

><fs> rsync-out /dest rsync://test.106.30:33335/dest/ archive:true 
><fs> q

$ ll hostdest/
total 0
drwxr-xr-x. 2 nobody nobody 30 May  6 10:34 dest
$ ll hostdest/dest/
total 0
-rw-r--r--. 1 nobody nobody 0 May  6 10:34 rsyncout_test.txt

Compared with the same files in step2, the permissions of them are different.
In summary, the permissions of the files copied out via rsync-out are not reasonable on fedora22.

Comment 6 Jaromír Cápík 2015-07-14 21:15:22 UTC
Hello. I'm facing exactly the same issue with rsync when syncing the newly generated filesystem content to / during the stage2 bootstrap with f21 sources. The issue was not present with f19 sources. So, it looks like something changed in rsync in the meantime.

Comment 7 Richard W.M. Jones 2015-07-14 21:39:52 UTC
(In reply to Jaromír Cápík from comment #6)
> Hello. I'm facing exactly the same issue with rsync when syncing the newly
> generated filesystem content to / during the stage2 bootstrap with f21
> sources. The issue was not present with f19 sources. So, it looks like
> something changed in rsync in the meantime.

I pretty much convinced myself that this was a problem in rsync.
In any case, libguestfs doesn't do anything more complex than
running rsync.  If you wish to debug this, you can see the exact
commands it runs in this file:

https://github.com/libguestfs/libguestfs/blob/master/daemon/rsync.c

Comment 8 Jaromír Cápík 2015-07-15 13:22:26 UTC
How about changing the component to rsync then?

Comment 9 Richard W.M. Jones 2015-07-15 14:08:34 UTC
Well we don't really understand what the bug is.  Moving it around
isn't going to help until someone does that work.

Comment 10 Jaromír Cápík 2015-07-15 14:32:23 UTC
In that case at least CCing rsync maintainers could help, right?

Comment 11 Jaromír Cápík 2015-07-15 14:39:19 UTC
NOTE: In my case the problem appears even when rsync-ing local directories only.

We do the following:

  rsync -av ./ /

Comment 12 Jaromír Cápík 2015-07-15 16:15:49 UTC
NOTE2: I just added --ignore-existing and it passed.

Comment 14 YongkuiGuo 2017-12-11 10:47:14 UTC
After running the libguestfs automated cases, all of the rsync-out cases failed with the following versions:

libguestfs-1.36.10-3.el7.x86_64
libguestfs-rsync-1.36.10-3.el7.x86_64
rsync-3.1.2-4.el7.x86_64


On rhel7.5 host and the user is 'root'

Steps:
1. Prepare the file on the host
# rm -fr /tmp/hostsrc
# rm -fr /tmp/hostdest
# mkdir /tmp/hostsrc
# mkdir /tmp/hostdest
# chmod 777 /tmp/hostdest
# touch /tmp/hostsrc/rsyncin_test.txt

2. Configure the rsyncd service.
# echo "port = 33335
pid file = rsyncd.pid
[src]
path = /tmp/hostsrc
comment = source
use chroot = false
read only = true
[dest]
path = /tmp/hostdest
comment = destination
use chroot = false
read only = false
" > /tmp/rsyncd.conf

# rsync --daemon --config=/tmp/rsyncd.conf

3. guestfish --network -a $GSTEST_image_path
><fs> run
><fs> mount-options noatime $mount_device /
><fs> rm-rf /dest
><fs> rm-rf /src
><fs> mkdir /src
><fs> mkdir /dest
><fs> touch /dest/rsyncout_test.txt
><fs> rsync-out /dest rsync://root@$ipaddr_of_host:33335/dest/ archive:true
libguestfs: error: rsync_out: '/dest' to 'rsync://root.144.58:33335/dest/': rsync: chgrp "dest" (in dest) failed: Operation not permitted (1)
rsync: chgrp "dest/.rsyncout_test.txt.d8OIl2" (in dest) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

So the same issue still exists in rsync version 3.1.2(#1432899).

Comment 15 YongkuiGuo 2018-01-10 05:33:40 UTC
Hi rjones,

I also think it's a problem in rsync with -a option(archive:true in guestfish).
-a option is a set of -rlptgoD. I had a try to use rsync command in local host.

e.g.
#guestmount -a misc.ext2.raw -m /dev/vol_test/vol_file /mnt/test
#cd /mnt/test
#mkdir dest
#touch dest/rsyncout_test.txt
#rsync -rlptoD dest rsync://root.73.37:33335/dest/

It works well. I just removed the -g option. The previous error also was displayed "chgrp ... failed" in guestfish according to the comment#14. I just want to inquire whether the -g option can be removed in libguestfs source code, in case we can work around this problem in libguestfs autotest.

Comment 23 YongkuiGuo 2018-01-16 11:41:39 UTC
(In reply to Michal Ruprich from comment #22)
> You should add the 'fake super = yes' option to the destination module. I
> didn't understand this at first as well so I asked the upstream some time
> ago and I received a nice answer here:
> 
> https://bugzilla.samba.org/show_bug.cgi?id=11414
> 
> Basically even if you run rsync as root it will not run under root when a
> transfer is initiated. So this error has nothing to do with SELinux but
> simply with the fact that only root is able to set group on a file.
> 
> Also if the original group from the sender side is missing in the conf file
> on receiving side, it will be set to nobody. You should add uid and gid
> options to the module. This is described in more detail in 'man rsyncd.conf'.
> 
> Hope this helps.
> 
> Regrads,
> Michal

It works fine with 'fake super = yes' option to the destination module. Thanks a lot.I will look at the rsyncd.conf carefully.