FWIW I cannot reproduce this with libguestfs-1.38.3-1.fc28.x86_64

I believe the error

tar -C /sysroot/ -cf - .
pclose: /: Success

comes from:

https://github.com/libguestfs/libguestfs/blob/51a9c874d3f0a9c4780f2cd3ee7072180446e685/daemon/tar.c#L387

It seems as if the tar command is exiting abruptly (without any
error message).

I wonder if the Ubuntu tar command has some sort of problem with
the CentOS filesystem.  SELinux labels?

You might try running:

$ virt-rescue --scratch

and checking if the tar command in the appliance is functioning properly.

Hi Richard. 

On Ubuntu 18.04, I receive the original error as well. Also, it gives same error when running against non-Centos images. For instance, the following also gives same error: 

$ virt-builder debian-8 -o debian8.img --no-network
$ mkdir debfs
$ virt-copy-out -a debian8.img / ~/debfs/


Follwing your suggestion, I've run 

$ virt-rescue --scratch 

And can confirm that tar is there and appears to be functional I only tested by tar-ing and untar-ing the included bin/ directory. (Not sure how I can access the debian8.img file as referenced above?)

However, one thing I notice from the debug log on my machine, are these lines:

guestfsd: main_loop: new request, len 0x34^M
libguestfs: trace: is_dir = 1
libguestfs: command: run: tar
libguestfs: command: run: \ -xf
libguestfs: command: run: \ -

It strikes me as odd that "tar -xf -" is being reported on 3 separate lines broken up like that...is it possible that its trying to run each as its own sub-command instead of running it as a single command?

(In reply to Barry S from comment #2)
> Hi Richard. 
> 
> On Ubuntu 18.04, I receive the original error as well. Also, it gives same
> error when running against non-Centos images. For instance, the following
> also gives same error: 
> 
> $ virt-builder debian-8 -o debian8.img --no-network
> $ mkdir debfs
> $ virt-copy-out -a debian8.img / ~/debfs/
> 
> 
> Follwing your suggestion, I've run 
> 
> $ virt-rescue --scratch 
> 
> And can confirm that tar is there and appears to be functional I only tested
> by tar-ing and untar-ing the included bin/ directory. (Not sure how I can
> access the debian8.img file as referenced above?)

Can you try:

$ virt-rescue --ro -a debian8.img

The image should be available as /dev/sda and can be mounted inside
virt-rescue, eg:

  ><rescue> mount /dev/sdaX /sysroot/

> However, one thing I notice from the debug log on my machine, are these
> lines:
> 
> guestfsd: main_loop: new request, len 0x34^M
> libguestfs: trace: is_dir = 1
> libguestfs: command: run: tar
> libguestfs: command: run: \ -xf
> libguestfs: command: run: \ -
> 
> It strikes me as odd that "tar -xf -" is being reported on 3 separate lines
> broken up like that...is it possible that its trying to run each as its own
> sub-command instead of running it as a single command?

No.

virt-copy-out works by running a copy of ‘tar -cf ...’ inside the appliance
and connecting it to a copy of ‘tar -xf ...’ running on the host.  The debug
messages above show the copy running on the host:

https://github.com/libguestfs/libguestfs/blob/51a9c874d3f0a9c4780f2cd3ee7072180446e685/lib/copy-in-out.c#L201
https://github.com/libguestfs/libguestfs/blob/51a9c874d3f0a9c4780f2cd3ee7072180446e685/lib/command.c#L426

>Can you try:
>
>$ virt-rescue --ro -a debian8.img
>
>The image should be available as /dev/sda and can be mounted inside
>virt-rescue, eg:
>
> <rescue> mount /dev/sdaX /sysroot/

I can mount without problem and see the files inside /sysroot

Any other test I can run? 

>No.

>virt-copy-out works by running a copy of ‘tar -cf ...’ inside the appliance
>and connecting it to a copy of ‘tar -xf ...’ running on the host.  The debug
>messages above show the copy running on the host

Thanks, understood.

I would try this test:

(1) Mount the root and maybe boot filesystems as they would
be in the real appliance, eg:

  ><rescue> mount /dev/sda3 /sysroot
  ><rescue> mount /dev/sda1 /sysroot/boot

(actual device names might be different)

(2) Run a tar command as the appliance does:

  ><rescue> tar -C /sysroot/ -cf - . >/dev/null

Does the tar command exit successfully?  Is any error printed?

Is the output (sent to /dev/null above) an actual tar file?

It's difficult to get coredumps inside virt-rescue but you may
be able to get a stack trace if tar is crashing by doing:

LD_PRELOAD=/lib/libSegFault.so tar [etc]

Thanks for the hand holding.

>(1) Mount the root and maybe boot filesystems as they would
>be in the real appliance, eg:
>
> <rescue> mount /dev/sda3 /sysroot
> <rescue> mount /dev/sda1 /sysroot/boot

Just a note. Appears this debian8.img file only has a single partition /dev/sda1 - therefore /boot is already there inside. 


>(2) Run a tar command as the appliance does:
>
>  <rescue> tar -C /sysroot/ -cf - . >/dev/null

Works fine with no errors. 

In addition, I ran:

<rescue> tar -C /sysroot/ -cf - . > myfile.tar
<rescue> file myfile.tar
myfile.tar: POSIX tar archive (GNU)


Additionally, I can run "tar tf myfile.tar" and successfully see all files contained inside.

Hi Richard and Barry,

I noticed that the following commands do not report any error (like Barry already mentioned)

# virt-rescue --ro -a debian8.img
><rescue> mount /dev/sda1 /sysroot
><rescue> tar -C /sysroot/ -cf - . > /dev/null

but guestfish fails

# guestfish
><fs> add-drive debian8.img
><fs> run
><fs> tar-out / - | tar tf -
libguestfs: error: file receive cancelled by daemon
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

What is the difference tar running with virt-rescue and when called from tar-out command of guestfish?

Is it possible to get a stack trace of tar when is executed by guestfish?

I managed to get strace log with this hack:

$ sudo mv /bin/tar /bin/tar-back
$ sudo cat > /bin/tar <<-EOF
#!/bin/bash
strace -o /tmp/strace.log /bin/tar-back $@
EOF
$ sudo chmod +x /bin/tar


This is the content of strace.log
https://paste.fedoraproject.org/paste/L5pqLmbtmYwNguWltZAARw/raw

I found the following description of the error message from tar

https://git.savannah.gnu.org/cgit/tar.git/tree/NEWS#n755

* Recognition of broken archives.

When supplied an archive smaller than 512 bytes in reading mode (-x,
-t), the previous version of tar silently ignored it, exiting with
code 0.  It is fixed.  Tar now issues the following diagnostic message:
'This does not look like a tar archive', and exits with code 2.

(In reply to rstoyanov1 from comment #7)
><rescue> tar -C /sysroot/ -cf - . > /dev/null
> 
> but guestfish fails
> 
> # guestfish
> ><fs> add-drive debian8.img
> ><fs> run
> ><fs> tar-out / - | tar tf -
> libguestfs: error: file receive cancelled by daemon
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> 

Thank you for this. 

I've downloaded libguestfs 1.39.9 from github and have built in a local directory on my Ubuntu 18.04 box. 

I can confirm that this version works with virt-builder / virt-copy-out without problems. So its definitely appears to be related to the version of guestfish that Ubuntu is shipping with (1.36.13). 

I'm hesitant to actually install this source version for production as there are so many dependencies related to libguestfs that it will cause a maintenance issues in trying to support. However, as a temporary workaround, I can just build my images in the 1.39.9 build directory and then move the resulting img-dir into the production area.

(In reply to Radostin Stoyanov from comment #8)
> I managed to get strace log with this hack:
> 
> $ sudo mv /bin/tar /bin/tar-back
> $ sudo cat > /bin/tar <<-EOF
> #!/bin/bash
> strace -o /tmp/strace.log /bin/tar-back $@
> EOF
> $ sudo chmod +x /bin/tar
> 
> 
> This is the content of strace.log
> https://paste.fedoraproject.org/paste/L5pqLmbtmYwNguWltZAARw/raw

This shows it is reading no data from stdin:

read(0, "", 10240)                      = 0

Since this bug seems to be specific to Ubuntu, it seems best to
continue this discussion on the downstream Launchpad bug:

https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1793056?comments=all

Agreed. Thanks for your help Richard.