Bug 562743 - autogen.sh: Failure to disable the use git and rsync (for offline bootstrap)
Summary: autogen.sh: Failure to disable the use git and rsync (for offline bootstrap)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: All
OS: All
low
low
Target Milestone: ---
Assignee: Eric Blake
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-02-08 09:21 UTC by Aleksey Avdeev
Modified: 2011-03-17 14:30 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-03-17 14:30:56 UTC


Attachments (Terms of Use)
libvirt-0.7.7-alt-offline-bootstrap.patch (5.98 KB, patch)
2010-03-20 07:35 UTC, Aleksey Avdeev
no flags Details | Diff
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch (1017 bytes, patch)
2010-03-20 22:34 UTC, Aleksey Avdeev
no flags Details | Diff
0002-Fix-status-function-in-initscript.patch (713 bytes, patch)
2010-03-20 22:36 UTC, Aleksey Avdeev
no flags Details | Diff
0002-autogen.sh-Always-call-the-bootstrap.patch (1.01 KB, patch)
2010-03-20 22:41 UTC, Aleksey Avdeev
no flags Details | Diff
0003-bootstrap.conf-Do-not-check-rsync-if-the-variable-is.patch (879 bytes, patch)
2010-03-20 22:42 UTC, Aleksey Avdeev
no flags Details | Diff
0004-README-hacking-If-you-do-not-need-to-download-po-fil.patch (850 bytes, patch)
2010-03-20 22:43 UTC, Aleksey Avdeev
no flags Details | Diff
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch (751 bytes, patch)
2010-07-22 07:46 UTC, Aleksey Avdeev
no flags Details | Diff
0002-bootstrap.conf-Do-not-check-rsync-if-the-variable-is.patch (707 bytes, patch)
2010-07-22 07:48 UTC, Aleksey Avdeev
no flags Details | Diff
0003-README-hacking-If-you-do-not-need-to-download-po-fil.patch (838 bytes, patch)
2010-07-22 07:50 UTC, Aleksey Avdeev
no flags Details | Diff
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch (820 bytes, patch)
2010-07-22 14:30 UTC, Aleksey Avdeev
no flags Details | Diff

Description Aleksey Avdeev 2010-02-08 09:21:42 UTC
Description of problem:

When using the bootstrap with the option --gnulib-srcdir=DIRNAME for the case when DIRNAME not git repo, returns error:

./bootstrap: invalid gnulib srcdir: <DIRNAME>


Version-Release number of selected component (if applicable):

libvirt v0.7.6 or earlier.


How reproducible:

100%


Steps to Reproduce:

1. Update submodule:

$ git submodule init
$ git submodule update

2. Create gnulib-libvirt.tar:

$ (cd .gnulib/ && git archive HEAD --prefix gnulib-libvirt/ ) > gnulib-libvirt.tar

3. Create gnulib-libvirt/ dir (unpacking gnulib-libvirt.tar):

$ tar xvf gnulib-libvirt.tar
gnulib-libvirt/
gnulib-libvirt/.cvsignore
gnulib-libvirt/.gitattributes
gnulib-libvirt/.gitignore
gnulib-libvirt/COPYING
...
gnulib-libvirt/users.txt

4. Run bootstrap:

$ ./bootstrap --gnulib-srcdir=gnulib-libvirt
./bootstrap: invalid gnulib srcdir: gnulib-libvirt
  
Actual results:

Eroor


Expected results:

Runs gnulib-tool...

Additional info:

I corrected a bug in the brunch ALT/fix/bootstrap repository git://git.altlinux.org/people/solo/packages/libvirt.git (see: http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=ALT/fix/bootstrap).

Comment 1 Aleksey Avdeev 2010-03-20 07:35:35 UTC
Created attachment 401399 [details]
libvirt-0.7.7-alt-offline-bootstrap.patch

git diff v0.7.7 ALT/fix/bootstrap

Comment 2 Aleksey Avdeev 2010-03-20 07:59:00 UTC
Comment on attachment 401399 [details]
libvirt-0.7.7-alt-offline-bootstrap.patch

  Реализация возможности офлайнового bootstrap для libvitr v0.7.7. Патчу соответствует бранч ALT/fix/bootstrap моего git repo (see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=shortlog;h=refs/heads/ALT/fix/bootstrap).

  Example of use in my spec (see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=blob;f=libvirt.spec;h=c72a672964e44c936f861a009abfdb7f3de5505a;hb=master):

%build
export GNULIB_SRCDIR=gnulib-%name-%version
export NO_GIT=t
export SKIP_PO=t
./autogen.sh

Comment 3 Aleksey Avdeev 2010-03-20 08:01:25 UTC
Comment on attachment 401399 [details]
libvirt-0.7.7-alt-offline-bootstrap.patch

  Implementation capacity offline bootstrap for libvitr v0.7.7. Patch against the brunch ALT/fix/bootstrap my git repo (see
http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=shortlog;h=refs/heads/ALT/fix/bootstrap).

  Example of use in my spec (see
http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=blob;f=libvirt.spec;h=c72a672964e44c936f861a009abfdb7f3de5505a;hb=master):

%build
export GNULIB_SRCDIR=gnulib-%name-%version
export NO_GIT=t
export SKIP_PO=t
./autogen.sh

Comment 4 Eric Blake 2010-03-20 12:03:36 UTC
I'm not sure why you insist on rerunning autogen.sh.  If you are building an rpm of libvirt, you should start from an existing tarball, which should already contain everything pre-boostrapped.  Therefore, building the rpm should start with './configure', not './autogen.sh'.  autogen.sh is intended for developers, not distributors, and it can be assumed that developers have git handy.

Comment 5 Aleksey Avdeev 2010-03-20 20:59:14 UTC
  To run autogen.sh no. He runs one time: only when building the package. (Only from the spec when I run rpmbuild -bi). So for me it is important that all he needed was already inside the src.rpm sent to the assembly.

  All source code in case src.rpm contains the results of such things as libtoolize, aclocal, autoconf, autoheader, automake and the like: all this is done already in the spec. These are the traditions of my distribution. (And it is easy: what to do by hand what you can instruct the robot?)

PS: I'm preparing a simplified patch.

Comment 6 Aleksey Avdeev 2010-03-20 22:29:40 UTC
Comment on attachment 401399 [details]
libvirt-0.7.7-alt-offline-bootstrap.patch

Branch ALT/fix/bootstrap updated, see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=shortlog;h=refs/heads/ALT/fix/bootstrap

Comment 7 Aleksey Avdeev 2010-03-20 22:34:14 UTC
Created attachment 401505 [details]
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch

bootstrap.conf: Do not add git in buildreq, if the variable is defined GNULIB_SRCDIR.

see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=1406a33820048789fee0e33b4e2165954446e4ee

Comment 8 Aleksey Avdeev 2010-03-20 22:36:28 UTC
Created attachment 401507 [details]
0002-Fix-status-function-in-initscript.patch

autogen.sh: Always call the bootstrap,
if you can not determine the hash submodule

see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=5497f7a05b61e925796d23c0ceaed15b61315924

Comment 9 Aleksey Avdeev 2010-03-20 22:40:26 UTC
Comment on attachment 401507 [details]
0002-Fix-status-function-in-initscript.patch

Этот патч добавлен по ошибочно.

Comment 10 Aleksey Avdeev 2010-03-20 22:41:46 UTC
Created attachment 401508 [details]
0002-autogen.sh-Always-call-the-bootstrap.patch

autogen.sh: Always call the bootstrap,
if you can not determine the hash submodule

see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=5497f7a05b61e925796d23c0ceaed15b61315924

Comment 11 Aleksey Avdeev 2010-03-20 22:42:42 UTC
Created attachment 401509 [details]
0003-bootstrap.conf-Do-not-check-rsync-if-the-variable-is.patch

bootstrap.conf: Do not check rsync, if the variable is defined SKIP_PO

see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=a82acee5d2b33d59219632d7c04d51ff4c2580b6

Comment 12 Aleksey Avdeev 2010-03-20 22:43:47 UTC
Created attachment 401510 [details]
0004-README-hacking-If-you-do-not-need-to-download-po-fil.patch

README-hacking: If you do not need to download po files,
define the variable SKIP_PO

see http://git.altlinux.org/people/solo/packages/?p=libvirt.git;a=commit;h=c1add6605849be4c88b80bd412355e280dbc783c

Comment 13 Daniel Berrangé 2010-07-21 16:52:01 UTC
Eric, does current libvirt GIT include these patches for bootstrap now ? I've lost track of what changes we made there

Comment 14 Eric Blake 2010-07-21 17:20:29 UTC
I still disagree with patches 1, 2, and 3 - you should NOT be running ./autogen.sh nor ./bootstrap if you are building from a tarball; so it should not matter what bootstrap.conf nor autogen.sh contains if you will not be running them.  Besides, what happens if your choice of $GNULIB_SRCDIR does not include the commit that libvirt is expecting, but you short-circuited things to not allow a git checkout to update to a newer GNULIB_SRCDIR?

./autogen.sh is intended solely for developers, not distro generation.

If you want to rerun the autotools in your spec file, run 'autoreconf -f', not './autogen.sh'.  But even then, I claim that you can get away without rebuilding the autotool files unless you are patching configure.ac or Makefile.am as one of your spec file patches.  At any rate, I claim that the bug is in the %build of your spec file, not in libvirt.

(If you can manage to convince me that patches 1 and 3 are still necessary, I'd rather see the style:

buildreq="
a
"
if cond; then
 buildreq="$buildreq
b
"
fi
buildreq="$buildreq
c
"

than the proposed

buildreq1="
a
"
buildreq2=
if cond; then
buildreq2="
b
"
buildreq="$buildreq1$buildreq2"

but that's stylistic, not a show-stopper)

Patch 4 is documentation only, and seems useful.  But it would be worth proposing that upstream to the bug-coreutils mailing list (libvirt borrows it's README-hacking from coreutils).

Aleksey, can we close this out as NOTABUG?

Comment 15 Aleksey Avdeev 2010-07-22 07:46:34 UTC
Created attachment 433627 [details]
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch

bootstrap.conf: Do not add git in buildreq, if the variable is defined GNULIB_SRCDIR.

see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=commit;h=cf117ebf0f91a4ac31d5c3eab616e5a9279facc5

Comment 16 Aleksey Avdeev 2010-07-22 07:48:37 UTC
Created attachment 433628 [details]
0002-bootstrap.conf-Do-not-check-rsync-if-the-variable-is.patch

bootstrap.conf: Do not check rsync, if the variable is defined SKIP_PO

see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=commit;h=987949fc2075ab8037daca509d2abd32c344e06c

Comment 17 Aleksey Avdeev 2010-07-22 07:50:05 UTC
Created attachment 433629 [details]
0003-README-hacking-If-you-do-not-need-to-download-po-fil.patch

README-hacking: If you do not need to download po files,
define the variable SKIP_PO

see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=commit;h=800beb0e1ecd2855edd044304bc297b9926028fe

Comment 18 Aleksey Avdeev 2010-07-22 07:52:06 UTC
Comment on attachment 401508 [details]
0002-autogen.sh-Always-call-the-bootstrap.patch

Obsolete in new ALT/fix/bootstrap

see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=shortlog;h=refs/heads/ALT/fix/bootstrap

Comment 19 Aleksey Avdeev 2010-07-22 11:32:55 UTC
1. I build the package is not from the official tarball, but directly from the git repository transferred to the build system.  All the necessary tarballs, sent directly to the assembly, build system creates itself, from these I commit.

2. Accessing external resources in the process of assembly (and / or at the preparatory stage tarballs build system), we considered a serious bug: all downloads must be fulfilled by man (in preparation for repository transferred to the assembly) and not a robot. Such is the policy of our distribution. 

2. In any case, I want to replace files autotool (autoconf, automake, libtool, etc.) distributive -- are with us are often patched.

  By virtue of paragraph 1 (source git and not the official tarball) in any case, I want to form the desired gnulib (=> run the bootstrap). It is logical to assign this work the robot (build system).

  Robot (build system) has no right to do git submodule update (Clause 2), but can generate tarball from a given commit and deploy it in the specified directory. If you transfer this directory through $ GNULIB_SRCDIR bootstrap works off without causing a git. (With me and you want: no need to put in git to build chroot.)

  If $GNULIB_SRCDIR used during assembly does not contain the right - build fails, you may need to correct his hands: build system should not perform such things as a git submodule update in Clause 2.

  Common sense patches bootstrap.conf - an exception to that requirement in the given conditions bootstrap does not actually use: git is not necessary when you specify $GNULIB_SRCDIR, but rsync is not necessary if you set $SKIP_PO. (If the original ootstrap.conf git and rsync always required.) I do not see that these patches can break. Important for me as functional, they add.

PS: I changed the style (patch for autogen.sh threw out - he no longer needed). See the new patches.

PPS: In my opinion this is an issue requiring correction.

Comment 20 Eric Blake 2010-07-22 12:40:13 UTC
Personally, I think your point 1 is a bit sadistic (in my opinion, a distro should be focused on stability, not bleeding edge - at which point, why not go with the stable tarball as the starting point), but I am not in a position to tell you that you can't choose to inflict this extra pain upon yourself (this is free software, after all).  So, given your point 1, that you would rather build from a git snapshot than from a release tarball, yes, you are now responsible for providing all the bits that would have been provided automatically by the release process - so you are now in the position of a developer, that must have all the development tools installed.

But I agree with point 2, where it logically follows that if you have already externally downloaded  all the bits in place, it should be possible to run ./bootstrap without having to download additional bits.

However, I still don't see why you don't want to have git installed.  For example, it is more than just ./bootstrap.  In my opinion, any sane distro will distribute the project's ChangeLog as part of the documentation of that package; but libvirt's ChangeLog is a generated file via build-aux/gitlog-to-changelog.  Are you accounting for that, and why not just have git installed to generate that changelog from your local git checkout of libvirt?  A git snapshot alone will not provide a correct ChangeLog - you would need the entire libvirt.git history.

Meanwhile, I think it would be possible to already have a copy of the upstream gnulib.git repository downloaded, then use some git config tricks, like adding this to your git config:

[url "path/to/local/gnulib"]
   insteadof = git://git.sv.gnu.org/gnulib.git

as well as having GNULIB_SRCDIR=path/to/local/gnulib in your environment, such that the action of 'git submodule init' will not cause any network traffic, even though git is still installed and run during ./bootstrap.  At which point, we didn't have to make any modifications to bootstrap, so I'm still not convinced that we need to remove git from bootstrap.conf.  And since it causes no network traffic, this would be no different than the fact that you require automake and autoconf to be installed, which are also prerequisites of ./bootstrap.

Now, on to rsync.  The latest bootstrap.conf _already_ includes SKIP_PO=true and omits a dependency on rsync, because we don't use the GNU translation project (according to git log, this was in 0.8.0).  So your patches are out of date - you shouldn't have to do anything to cripple rsync, because there is nothing to cripple.

Comment 21 Eric Blake 2010-07-22 12:54:14 UTC
A couple more points:

If we end up agreeing that bootstrap should not require git under some circumstances, that decision must be based on something _other_ than GNULIB_SRCDIR.  As a developer, I _always_ have GNULIB_SRCDIR set, even though I still expect git to query the network, because I rely on the reduction in download time and disk space used by having a local reference repository to pre-seed my download.  In other words, GNULIB_SRCDIR is an optimization hint (as of 0.7.7), rather than an alternate upstream location; so to claim that you don't need git, you would want to be telling ./bootstrap that you have an alternate upstream location that exists with all the bits downloaded from the exact commit that you would otherwise be downloading from upstream gnulib.git.

Also, ./bootstrap is maintained by gnulib.  Maybe it would make sense asking bug-gnulib@gnu.org to add an upstream feature SKIP_GIT (comparable to SKIP_PO), as a way of removing the git dependency without relying on writing shell conditionals in ./bootstrap.conf.

Comment 22 Aleksey Avdeev 2010-07-22 13:56:59 UTC
Comment on attachment 433628 [details]
0002-bootstrap.conf-Do-not-check-rsync-if-the-variable-is.patch

Obsolete with v0.8.0

Comment 23 Aleksey Avdeev 2010-07-22 13:57:17 UTC
Comment on attachment 433629 [details]
0003-README-hacking-If-you-do-not-need-to-download-po-fil.patch

Obsolete with v0.8.0

Comment 24 Aleksey Avdeev 2010-07-22 14:30:13 UTC
Created attachment 433717 [details]
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch

bootstrap.conf: Do not add git in buildreq, if the variable GNULIB_SRCDIR is defined and dir $GNULIB_SRCDIR/.git exist.

see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=commit;h=bc91ac62006cb7dc33d88ff0333a4cb36b4518de

Comment 25 Aleksey Avdeev 2010-07-22 14:32:05 UTC
Comment on attachment 433627 [details]
0001-bootstrap.conf-Do-not-add-git-in-buildreq-if-the-var.patch

Obsolete, see http://git.altlinux.org/people/solo/packages/libvirt.git?p=libvirt.git;a=commit;h=bc91ac62006cb7dc33d88ff0333a4cb36b4518de

Comment 26 Aleksey Avdeev 2010-07-22 18:34:53 UTC
  I offered to bug-gnulib@gnu.org patch that adds a build-aux/bootstrap
handling variable $NO_GIT and parameter --no-git (see
http://lists.gnu.org/archive/html/bug-gnulib/2010-03/msg00229.html). Then it
died down. I can propose it again, replacing the variable on $SKIP_GIT, and the
option to --scip-git, but I doubt that this proposal will be accepted - all
functionality is already there:

1. In comments to the parameter - gnulib-srcdir said that work without recourse
to external resources is possible. Quote build-aux/bootstrap (./bootstrap):

 --gnulib-srcdir=DIRNAME  Specify the local directory where gnulib
                          sources reside.  Use this if you already
                          have gnulib sources on your machine, and
                          do not want to waste your bandwidth downloading
                          them again.  Defaults to \$GNULIB_SRCDIR.

2. Quote build-aux/bootstrap (./bootstrap):

for option
do
  case $option in
...
  --gnulib-srcdir=*)
    GNULIB_SRCDIR=`expr "X$option" : 'X--gnulib-srcdir=\(.*\)'`;;
...

# Get gnulib files.

case ${GNULIB_SRCDIR--} in
...
*)
  # Use GNULIB_SRCDIR as a reference.
  if test -d "$GNULIB_SRCDIR"/.git && \
        git_modules_config submodule.gnulib.url >/dev/null; then
    echo "$0: getting gnulib files..."
...
  fi
  ;;
esac

gnulib_tool=$GNULIB_SRCDIR/gnulib-tool
<$gnulib_tool || exit

  That is, if $GNULIB_SRCDIR determined and he did not git repository - git to
update the contents of $GNULIB_SRCDIR not used and the variable itself does not
change.

  In cases of libvirt this idyll is disturbed only by a direct indication git
variable buildreq file ./bootstrap.conf. What I propose to fix their patch.

PS: The condition does not add git clarified.

Comment 27 Eric Blake 2010-07-26 23:40:58 UTC
I'd like to add bug-gnulib@gnu.org to the cc of this bug, but bugzilla won't allow that - so you'll have to repeat the key points of this bug to the upstream gnulib list as part of (re-)proposing your patch from March.

You've made a few additional points in this bug that you didn't make back in March, so there is a good chance that you can still get an upstream patch to gnulib's ./bootstrap to support a --skip-git or --no-git option that is then smart enough to override any bootstrap.conf dependency on git.  But I do insist that it be an explicit option (to make it clear to the person using a static checkout that the onus is on them, now that git is no longer able to use submodules to guarantee the correct checkout), and not just rely on whether $GNULIB_SRCDIR points to a git repository.  Furthermore, I still insist that any changes should be in bootstrap, so that all projects sharing that file reap the benefits, rather than adding logic to bootstrap.conf which would have to be duplicated to each client project.

For that matter, the code you quoted has a bug - a bare git repository does not have $GNULIB_SRCDIR/.git, but can (and should) still be usable as a submodule seed.

If upstream gnulib accepts a patch that is sufficient for your needs, then this BZ serves as the justification for me to immediately port it to libvirt (by updating the gnulib submodule to an appropriate point).  But right now, this bug is looking like it is conditional on upstream gnulib reaction (last time around, I was the only person who replied on the gnulib list, and your answers back then were not as detailed as what is in this bug report, so I didn't see any reason to take action back then).

Comment 28 Aleksey Avdeev 2010-08-09 06:28:14 UTC
Patch to gnulib proposed, see http://lists.gnu.org/archive/html/bug-gnulib/2010-08/msg00047.html

Comment 29 Eric Blake 2011-01-20 20:58:19 UTC
I guess I dragged my feet on this long enough that Benjamin Lindner stepped up and did it instead:
http://lists.gnu.org/archive/html/bug-gnulib/2011-01/msg00284.html

At any rate, here's the proposed libvirt patch that reflects that:
https://www.redhat.com/archives/libvir-list/2011-January/msg00873.html

Comment 30 Cole Robinson 2011-03-17 14:30:56 UTC
Patch upstream now:

http://libvirt.org/git/?p=libvirt.git;a=commit;h=125978fe3baba34959185f94c14a2a5e05c4a28c

Sounds like bug can be closed? Please reopen if I'm wrong


Note You need to log in before you can comment on or make changes to this bug.