+++ This bug was initially created as a clone of Bug #1091856 +++

Description of problem:
scrub-file can't handle the link file, failed by "no rw access"


Version-Release number of selected component (if applicable):
1.27.11


How reproducible:
100%


Steps to Reproduce:
# guestfish -x -v -N fs -m /dev/sda1 write /test.log "hello" : ln-s /test.log /link.log : scrub-file /link.log

# [omitted logs]
libguestfs: trace: mount_options = 0
libguestfs: trace: write "/test.log" "hello"
libguestfs: trace: internal_write "/test.log" "hello"
guestfsd: main_loop: proc 74 (mount_options) took 0.00 seconds
guestfsd: main_loop: new request, len 0x44
guestfsd: main_loop: proc 246 (internal_write) took 0.00 seconds
libguestfs: trace: internal_write = 0
libguestfs: trace: write = 0
libguestfs: trace: ln_s "/test.log" "/link.log"
guestfsd: main_loop: new request, len 0x48
ln -s -- /test.log /sysroot/link.log
guestfsd: main_loop: proc 166 (ln_s) took 0.00 seconds
libguestfs: trace: ln_s = 0
libguestfs: trace: scrub_file "/link.log"
guestfsd: main_loop: new request, len 0x38
scrub -r /sysroot/link.log
scrub: no rw access to /sysroot/link.log
guestfsd: error: /link.log: scrub: no rw access to /sysroot/link.log
guestfsd: main_loop: proc 115 (scrub_file) took 0.00 seconds
libguestfs: trace: scrub_file = -1 (error)
libguestfs: error: scrub_file: /link.log: scrub: no rw access to /sysroot/link.log
libguestfs: trace: close
libguestfs: closing guestfs handle 0x7fe8766d7730 (state 2)
libguestfs: trace: internal_autosync
guestfsd: main_loop: new request, len 0x28
umount /sysroot
fsync /dev/sda
guestfsd: main_loop: proc 282 (internal_autosync) took 0.03 seconds
libguestfs: trace: internal_autosync = 0
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsLzMSKl


Actual results:
libguestfs: error: scrub_file: /link.log: scrub: no rw access to /sysroot/link.log


Expected results:
scrub-file can handle link file like linux original command 'scrub'


Additional info:
Same problem exist in rhel6, libguestfs-1.20.11-2.el6.x86_64

# guestfish -N fs -m /dev/sda1 write /test.log "hello" : ln-s /test.log /link.log : scrub-file /link.log
libguestfs: error: scrub_file: /link.log: scrub: /sysroot/link.log does not exist

--- Additional comment from Richard W.M. Jones on 2014-04-28 11:10:39 CEST ---

This is a tricky bug.  I'll make a note here of why it happens and
why it's not that easy to solve:

Why it happens
--------------

You are creating a symlink in the target filesystem which looks
like this:

  /link.log -> /test.log

However when libguestfs mounts the target filesystem in the appliance,
it does so under a directory called /sysroot.  So you end up with
a link which looks like this:

  /sysroot/link.log -> /test.log

When we run the scrub command (from the appliance) we run it on the
sysroot path, ie:

  /usr/bin/scrub /sysroot/link.log

and of course this doesn't work because /sysroot/link.log is a broken
link.

There are quite a lot of API calls which fail in the same way.
See also: bug 604041.

Why it's not easy to fix
------------------------

One obvious fix would be to run scrub in a chroot in /sysroot.  This
doesn't work because the scrub binary (and any other files it may need)
are only available in the appliance, not in the chroot.

Another possible fix would be to resolve symlinks in the daemon when
we modify the path to add /sysroot in front.  This is technically quite
complex because we're essentially duplicating the kernel's own code and
the benefit of libguestfs is it *doesn't* duplicate kernel code!  In
particular this wouldn't work if there are filesystems which have magic
symlink behaviour (I'm looking at you, ntfs-3g).

I think a better fix is going to be something like this:

  (1) Fork a process which chroots and resolves the symlink using the kernel.

  (2) Forked subprocess opens the target file and passes the file descriptor
      back to the main process (eg. using SCM_RIGHTS).

  (3) Main process runs `scrub /dev/fd/<N>'.

I'm open to suggestions about an easier way to solve this reliably.

--- Additional comment from Daniel Berrange on 2014-04-28 11:17:57 CEST ---

How about have the appliance filesystem at /.appliance and the guest filesystem at /.  That way absolute links in the guest FS "just work".  The obvious problem with this is the bootstrapping one - how to get the appliance to be at /.appliance since it needs to be present before you've actually setup / with the guest FS. I don't know if its possible to pivot_root ?

Alternatively perhaps have the / be the appliance, /sysroot be the guest, and then bind mount / to /sysroot/.appliance. Then when you chroot to run scrub, you can still access it as /.appliance/bin/scrub ?

--- Additional comment from Richard W.M. Jones on 2014-04-28 11:20:21 CEST ---

(In reply to Daniel Berrange from comment #2)
> How about have the appliance filesystem at /.appliance and the guest
> filesystem at /.  That way absolute links in the guest FS "just work".  The
> obvious problem with this is the bootstrapping one - how to get the
> appliance to be at /.appliance since it needs to be present before you've
> actually setup / with the guest FS. I don't know if its possible to
> pivot_root ?
> 
> Alternatively perhaps have the / be the appliance, /sysroot be the guest,
> and then bind mount / to /sysroot/.appliance. Then when you chroot to run
> scrub, you can still access it as /.appliance/bin/scrub ?

I guess the problem is that scrub needs to load other files
(eg. libraries, MO files, etc) from the regular paths.

--- Additional comment from Daniel Berrange on 2014-04-28 11:28:21 CEST ---

(In reply to Richard W.M. Jones from comment #3)
> (In reply to Daniel Berrange from comment #2)
> > How about have the appliance filesystem at /.appliance and the guest
> > filesystem at /.  That way absolute links in the guest FS "just work".  The
> > obvious problem with this is the bootstrapping one - how to get the
> > appliance to be at /.appliance since it needs to be present before you've
> > actually setup / with the guest FS. I don't know if its possible to
> > pivot_root ?
> > 
> > Alternatively perhaps have the / be the appliance, /sysroot be the guest,
> > and then bind mount / to /sysroot/.appliance. Then when you chroot to run
> > scrub, you can still access it as /.appliance/bin/scrub ?
> 
> I guess the problem is that scrub needs to load other files
> (eg. libraries, MO files, etc) from the regular paths.

For libraries there's LD_LIBRARY_PATH, but there's no equiv for MO files AFAIK - wonder if that's a worthwhile RFE to glibc since it presumably hurts people already if they install translated apps into --prefix=$HOME/usr or similar