Bug 507074 - removing files using wildcard isn't working
Summary: removing files using wildcard isn't working
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libguestfs
Version: unspecified
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-06-20 14:36 UTC by Marek Goldmann
Modified: 2010-03-27 10:26 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-27 10:26:57 UTC
Embargoed:


Attachments (Terms of Use)

Description Marek Goldmann 2009-06-20 14:36:12 UTC
Description of problem:

I think there is a bug in "rm-rf" (possibly "rm" and "command" too) command. If I try to remove some files using a wildcard, none of those files is removed.

Version-Release number of selected component (if applicable):

[oddthesis@bolek-f11 ~]$ rpm -qa | grep libguest
libguestfs-1.0.21-3.fc11.x86_64
libguestfs-java-devel-1.0.21-3.fc11.x86_64
libguestfs-devel-1.0.21-3.fc11.x86_64
libguestfs-java-1.0.21-3.fc11.x86_64
libguestfs-javadoc-1.0.21-3.fc11.x86_64
ocaml-libguestfs-devel-1.0.21-3.fc11.x86_64
python-libguestfs-1.0.21-3.fc11.x86_64
ocaml-libguestfs-1.0.21-3.fc11.x86_64
perl-libguestfs-1.0.21-3.fc11.x86_64
ruby-libguestfs-1.0.21-3.fc11.x86_64

[oddthesis@bolek-f11 ~]$ rpm -qa | grep guestfish
guestfish-1.0.21-3.fc11.x86_64

Steps to Reproduce:
1. Run guestfish, mount image, launch
2. Create some files
3. Try to remove using rm-rf with wildcard (e.g.: rm-rf /home/thin/as*)
  
Actual results:

Files aren't removed.

Expected results:

Files are removed.

Additional info:

Session from guestfish:

><fs> ls /home/thin
.bash_logout
.bash_profile
.bashrc
as
asbvc
asbvcsdf
><fs> rm-rf /home/thin/as*
><fs> ls /home/thin
.bash_logout
.bash_profile
.bashrc
as
asbvc
asbvcsdf
><fs> command "rm -rf /home/thin/as*"

><fs> ls /home/thin
.bash_logout
.bash_profile
.bashrc
as
asbvc
asbvcsdf

Comment 1 Richard W.M. Jones 2009-06-20 15:14:47 UTC
Acknowledged.

The reason for this is because we don't use the shell to
run the 'rm -rf' command (which would expand wildcards if
we did it).  Instead we run the rm -rf command directly.
No shell involved, so no wildcards are expanded.

It's a bit tricky to know what the Right Thing to do here is.
Either I change the contract of the guestfs_rm_rf [1] so that
it explicitly promises to expand wildcards.

Or we expand wildcards in guestfish.

Or we add another command which expands wildcards.

Or we do nothing and close/wontfix.

I'm not sure which to do yet!

[1] http://libguestfs.org/guestfs.3.html#guestfs_rm_rf

Comment 2 Richard W.M. Jones 2009-06-20 15:18:10 UTC
Additional: If you are using a language binding instead
of guestfish, you can of course simulate what you want
easily.  Just call the 'ls' command first, filter out the
files that you want to remove, and then call 'rm' on each
one.

http://libguestfs.org/guestfs.3.html#guestfs_ls
http://libguestfs.org/guestfs.3.html#guestfs_rm

I'm beginning to think that the right thing to do
here is for guestfish to expand wildcards, although
that will break other things in a subtle way (eg.
should it expand them into additional parameters,
or into a list of strings, and what about commands
which don't take lists of strings).  Knotty.

Comment 3 Marek Goldmann 2009-06-20 15:40:22 UTC
(In reply to comment #2)
> Additional: If you are using a language binding instead
> of guestfish, you can of course simulate what you want
> easily.  Just call the 'ls' command first, filter out the
> files that you want to remove, and then call 'rm' on each
> one.

Yah, that's an option to do this, but this is somewhat confusing for people I think. I just read man rm to check if I'm doing everything good :)

> 
> http://libguestfs.org/guestfs.3.html#guestfs_ls
> http://libguestfs.org/guestfs.3.html#guestfs_rm
> 
> I'm beginning to think that the right thing to do
> here is for guestfish to expand wildcards, although
> that will break other things in a subtle way (eg.
> should it expand them into additional parameters,
> or into a list of strings, and what about commands
> which don't take lists of strings).  Knotty.  

I don't know if fixing this _only_ in guestfish is a right way.

I see that this is more general problem I thought before. What with mv and cp commands? Now it's somewhat not intuitive for users.

Maybe expanding wildcards should be implemented in all file functions by default? It is possible?

There will be still pb with using "command" command, but a mention in documentation should be sufficient.

Comment 4 Richard W.M. Jones 2009-06-21 12:14:21 UTC
(In reply to comment #3)
> > I'm beginning to think that the right thing to do
> > here is for guestfish to expand wildcards, although
> > that will break other things in a subtle way (eg.
> > should it expand them into additional parameters,
> > or into a list of strings, and what about commands
> > which don't take lists of strings).  Knotty.  
> 
> I don't know if fixing this _only_ in guestfish is a right way.
> 
> I see that this is more general problem I thought before. What with mv and cp
> commands? Now it's somewhat not intuitive for users.
> 
> Maybe expanding wildcards should be implemented in all file functions by
> default? It is possible?

It's not appropriate for C API calls to take wildcards by default.  No
other API calls work like this in Unix.  In Unix shells expand wildcards.

guestfish doesn't expand wildcards, and is not much more than a
dumb wrapper around the C API calls, hence the mismatch and
user confusion here.

Unfortunately adding wildcard expansion to guestfish is also not
simple without reworking the code which generates API calls and all
the language bindings, so they can take lists of strings in place of
simple strings.

Might consider an explicit loop construct and glob expansion in
guestfish - something like:

 foreach glob(/home/foo/*) rm $$

but with a nicer syntax.  Interestingly this is how early Unix shells
worked too, before such issues were properly resolved.

> There will be still pb with using "command" command, but a mention in
> documentation should be sufficient.  

Agreed - we should probably add a 'shell' call where it is explicit
that the parameters are passed to a shell, wildcards are expanded
and so on.  However note that 'command', 'shell' etc are only useful
where you are editing a Linux operating system image.  They don't
work on empty images, etc.

Comment 5 Marek Goldmann 2009-06-21 13:00:40 UTC
(In reply to comment #4)
> It's not appropriate for C API calls to take wildcards by default.  No
> other API calls work like this in Unix.  In Unix shells expand wildcards.

OK.

> guestfish doesn't expand wildcards, and is not much more than a
> dumb wrapper around the C API calls, hence the mismatch and
> user confusion here.
> 
> Unfortunately adding wildcard expansion to guestfish is also not
> simple without reworking the code which generates API calls and all
> the language bindings, so they can take lists of strings in place of
> simple strings.
> 
> Might consider an explicit loop construct and glob expansion in
> guestfish - something like:
> 
>  foreach glob(/home/foo/*) rm $$
> 
> but with a nicer syntax.  Interestingly this is how early Unix shells
> worked too, before such issues were properly resolved.
> 
> > There will be still pb with using "command" command, but a mention in
> > documentation should be sufficient.  
> 
> Agreed - we should probably add a 'shell' call where it is explicit
> that the parameters are passed to a shell, wildcards are expanded
> and so on.  However note that 'command', 'shell' etc are only useful
> where you are editing a Linux operating system image.  They don't
> work on empty images, etc.  

Adding 'shell' command sounds good and this could be the best solution for that pb. Shell could run only on Linux (name suggest that). This and difference between 'command' command should be also documented very well to avoid many user questions I think.

I've tried to simulate a shell command using:

><fs> command "/bin/bash -c 'ls -all /'"
libguestfs: error: -all: -c: line 0: unexpected EOF while looking for matching `''
-all: -c: line 1: syntax error: unexpected end of file

Is there a way to invoke a command using bash with current libguestfs?

Comment 6 Richard W.M. Jones 2009-06-21 13:13:46 UTC
On the last point, see bug 503134.

Comment 7 Richard W.M. Jones 2009-06-22 06:51:37 UTC
This cset adds support for 'sh' and 'sh-lines' commands:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=57d2dfab18ad3d987d9273bb7c1f42e73e0bbcb2

Example usage:

 ><fs> sh "echo /s*"
 /sbin /selinux /srv /sys
 
 ><fs> sh-lines "ls -1d /s*"
 /sbin
 /selinux
 /srv
 /sys

Comment 8 Richard W.M. Jones 2009-06-22 08:40:52 UTC
This cset adds support for the guestfish 'glob' command
which can be used to support wildcards for existing
commands:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=394c8bec21d47b74567a4148fdbf87318c301441

The documentation describes this command as follows:

    WILDCARDS AND GLOBBING
       Neither guestfish nor the underlying guestfs API performs wildcard
       expansion (globbing) by default.  So for example the following will not
       do what you expect:

        rm-rf /home/*

       Assuming you don’t have a directory literally called "/home/*" then the
       above command will return an error.

       To perform wildcard expansion, use the "glob" command.

        glob rm-rf /home/*

       runs "rm-rf" on each path that matches (ie. potentially running the
       command many times), equivalent to:

        rm-rf /home/jim
        rm-rf /home/joe
        rm-rf /home/mary

       "glob" only works on simple guest paths and not on device names.

       If you have several parameters, each containing a wildcard, then glob
       will perform a cartesian product.

I will consider this bug now done.  Please reset
the bug status to 'ASSIGNED' with an explanation if this
is not sufficient for your needs.

Comment 9 Marek Goldmann 2009-06-22 09:01:23 UTC
(In reply to comment #7)
> This cset adds support for 'sh' and 'sh-lines' commands:
> 
> http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=57d2dfab18ad3d987d9273bb7c1f42e73e0bbcb2
> 
> Example usage:
> 
>  ><fs> sh "echo /s*"
>  /sbin /selinux /srv /sys
> 
>  ><fs> sh-lines "ls -1d /s*"
>  /sbin
>  /selinux
>  /srv
>  /sys  

Great, exactly something like that I was looking for!

Can you provide a package for Fedora 11 so I can begin to use it?

Comment 10 Fedora Update System 2009-06-23 09:15:01 UTC
libguestfs-1.0.51-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.51-1.fc11

Comment 11 Fedora Update System 2009-06-29 19:27:41 UTC
libguestfs-1.0.54-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.54-2.fc11

Comment 12 Fedora Update System 2009-07-02 23:26:28 UTC
libguestfs-1.0.55-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.55-1.fc11

Comment 13 Fedora Update System 2009-07-11 09:27:26 UTC
libguestfs-1.0.58-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.58-2.fc11

Comment 14 Fedora Update System 2009-07-11 17:12:48 UTC
libguestfs-1.0.54-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2009-07-19 10:28:03 UTC
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Richard W.M. Jones 2010-03-27 10:26:57 UTC
Long fixed upstream with the 'glob' command, see
comment 8.


Note You need to log in before you can comment on or make changes to this bug.