Bug 1195881 - virt-builder can't docker pull
Summary: virt-builder can't docker pull
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libguestfs
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-24 19:01 UTC by James (purpleidea)
Modified: 2018-07-18 14:35 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)

Description James (purpleidea) 2015-02-24 19:01:49 UTC
Description of problem:

You can't 'docker pull' in a virt-builder environment. I suspect this is because something needed by the docker daemon is missing in the virt-builder environment.

As an aside, I think it's ridiculous that you need a daemon running to do a docker pull, but fixing that is out of scope for this bug report :(


Version-Release number of selected component (if applicable):
$ virt-builder --version
virt-builder 1.28.6

(on Fedora 21)

How reproducible:
100%

Steps to Reproduce:

$ virt-builder fedora-21 --install docker --run-command 'docker pull fedora:21'
[   8.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   8.0] Planning how to build this image
[   8.0] Uncompressing
[  25.0] Opening the new disk
[  60.0] Setting a random seed
[  60.0] Installing packages: docker
[ 154.0] Running: docker pull fedora:21
ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, '')
Trying other mirror.
Resolving Dependencies
--> Running transaction check
---> Package docker-io.x86_64 0:1.5.0-1.fc21 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package           Arch           Version                 Repository       Size
================================================================================
Installing:
 docker-io         x86_64         1.5.0-1.fc21            updates         5.8 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 5.8 M
Installed size: 26 M
Downloading packages:
warning: /var/cache/yum/x86_64/21/updates/packages/docker-io-1.5.0-1.fc21.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY
Public key for docker-io-1.5.0-1.fc21.x86_64.rpm is not installed
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64
Importing GPG key 0x95A43F54:
 Userid     : "Fedora (21) <fedora>"
 Fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54
 Package    : fedora-repos-21-2.noarch (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-io-1.5.0-1.fc21.x86_64                                1/1 
  Verifying  : docker-io-1.5.0-1.fc21.x86_64                                1/1 

Installed:
  docker-io.x86_64 0:1.5.0-1.fc21                                               

Complete!
time="2015-02-24T13:49:03-05:00" level="fatal" msg="Post http:///var/run/docker.sock/v1.17/images/create?fromImage=fedora%3A21: dial unix /var/run/docker.sock: no such file or directory" 
virt-builder: error: docker pull fedora:21: command exited with an error

If reporting bugs, run virt-builder with debugging enabled and include the 
complete output:

  virt-builder -v -x [...]



Actual results:
fail.

Expected results:
win.

Additional info:
I didn't attach the long debug output because this is an easy reproducer. If you can't reproduce, let me know, and I'll attach. Cheers!

Comment 2 Richard W.M. Jones 2015-03-04 15:08:05 UTC
The following *nearly* worked for me (and yes, the 'sleep' command
*is* necessary, grumble grumble):

virt-builder fedora-21 \
    --install docker \
    --run-command '
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
    '

It eventually prints:
[...]
511136ea3c5a: Pulling fs layer
00a0c78eeb6d: Pulling fs layer
834629358fe2: Pulling fs layer
511136ea3c5a: Verifying Checksum
511136ea3c5a: Download complete
00a0c78eeb6d: Verifying Checksum
00a0c78eeb6d: Download complete
511136ea3c5a: Pull complete
00a0c78eeb6d: Pull complete

followed by an error, presumably because 'docker pull' exits
with the wrong error code, or doesn't print an error message.
You could add ||: after the docker pull command to ignore that.

Proxy settings from outside virt-builder are, however, not
passed through to docker, although as far as I can see that
is a docker problem, not a virt-builder problem.  Similar
to: https://github.com/tmatilai/vagrant-proxyconf/issues/97

Comment 3 James (purpleidea) 2015-03-08 03:48:21 UTC
Hrm. I ran that and here is what I got:

$ virt-builder fedora-21 \
>     --install docker \
>     --run-command '
>       docker -H unix:///var/run/docker.sock -d &
>       sleep 5
>       docker pull fedora:21
>     '

[   4.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   5.0] Planning how to build this image
[   5.0] Uncompressing
[  17.0] Opening the new disk
[  40.0] Setting a random seed
[  40.0] Installing packages: docker
[ 102.0] Running: 
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
    
[ 621.0] Setting passwords
virt-builder: Setting random password of root to 7II5KPoA7uHByrt1
[ 622.0] Finishing off
virt-builder: error: libguestfs error: umount_all: umount: /sysroot: 
umount: /sysroot: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

If reporting bugs, run virt-builder with debugging enabled and include the 
complete output:

  virt-builder -v -x [...]



I didn't dig too deeply, but is this sysroot error caused by something that the daemon is still holding on to? What about the sleep 5, why is this necessary? Is there a clean solution, or a bug I can file with the docker people?

Cheers

Comment 4 Richard W.M. Jones 2015-03-08 13:31:46 UTC
Yup I suspect the docker daemon is holding /sysroot open.  Probably
adding `killall docker' as the last command could fix that.  Or
sending a "terminate the daemon" command if there is one.

The sleep 5 was necessary for me - it could be that docker doesn't
create the socket before forking into the background (which would be
a bug in docker if true).

Comment 5 James (purpleidea) 2015-03-08 20:29:37 UTC
I've created a docker issue to see if those folks can help suggest a solution or a fix, so that this works together more elegantly.

https://github.com/docker/docker/issues/11241

Comment 6 James (purpleidea) 2015-03-08 20:52:18 UTC
$ cat vb.sh
#!/bin/bash

virt-builder fedora-21 \
    --install docker \
    --run-command '
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
      killall docker
    '

$ ./vb.sh 
[   2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   2.0] Planning how to build this image
[   2.0] Uncompressing
[  14.0] Opening the new disk
[  29.0] Setting a random seed
[  29.0] Installing packages: docker
[  62.0] Running: 
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
      killall docker
    
ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, '')
Trying other mirror.
Resolving Dependencies
--> Running transaction check
---> Package docker-io.x86_64 0:1.5.0-1.fc21 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package           Arch           Version                 Repository       Size
================================================================================
Installing:
 docker-io         x86_64         1.5.0-1.fc21            updates         5.8 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 5.8 M
Installed size: 26 M
Downloading packages:
warning: /var/cache/yum/x86_64/21/updates/packages/docker-io-1.5.0-1.fc21.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY
Public key for docker-io-1.5.0-1.fc21.x86_64.rpm is not installed
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64
Importing GPG key 0x95A43F54:
 Userid     : "Fedora (21) <fedora>"
 Fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54
 Package    : fedora-repos-21-2.noarch (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-io-1.5.0-1.fc21.x86_64                                1/1 
  Verifying  : docker-io-1.5.0-1.fc21.x86_64                                1/1 

Installed:
  docker-io.x86_64 0:1.5.0-1.fc21                                               

Complete!
time="2015-03-08T16:34:31-04:00" level="info" msg="+job serveapi(unix:///var/run/docker.sock)" 
time="2015-03-08T16:34:31-04:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)" 
time="2015-03-08T16:34:31-04:00" level="warning" msg="WARNING: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors" 
time="2015-03-08T16:34:32-04:00" level="info" msg="+job init_networkdriver()" 
time="2015-03-08T16:34:32-04:00" level="info" msg="-job init_networkdriver() = OK (0)" 
time="2015-03-08T16:34:32-04:00" level="info" msg="WARNING: mountpoint for memory not found\n" 
time="2015-03-08T16:34:32-04:00" level="info" msg="Loading containers: start." 

time="2015-03-08T16:34:32-04:00" level="info" msg="Loading containers: done." 
time="2015-03-08T16:34:32-04:00" level="info" msg="docker daemon: 1.5.0 a8a31ef/1.5.0; execdriver: native-0.2; graphdriver: devicemapper" 
time="2015-03-08T16:34:32-04:00" level="info" msg="+job acceptconnections()" 
time="2015-03-08T16:34:32-04:00" level="info" msg="-job acceptconnections() = OK (0)" 
time="2015-03-08T16:34:36-04:00" level="info" msg="POST /v1.17/images/create?fromImage=fedora%3A21" 
time="2015-03-08T16:34:36-04:00" level="info" msg="+job pull(fedora, 21)" 
time="2015-03-08T16:34:36-04:00" level="info" msg="+job resolve_repository(fedora)" 
time="2015-03-08T16:34:36-04:00" level="info" msg="-job resolve_repository(fedora) = OK (0)" 
time="2015-03-08T16:34:36-04:00" level="info" msg="+job trust_update_base()" 
time="2015-03-08T16:34:36-04:00" level="info" msg="-job trust_update_base() = OK (0)" 
time="2015-03-08T16:34:37-04:00" level="info" msg="+job trust_key_check(/library/fedora)" 
time="2015-03-08T16:34:37-04:00" level="info" msg="-job trust_key_check(/library/fedora) = OK (0)" 
time="2015-03-08T16:34:37-04:00" level="info" msg="Image manifest for fedora:21 has been verified" 
511136ea3c5a: Pulling fs layer
00a0c78eeb6d: Pulling fs layer
834629358fe2: Pulling fs layer
511136ea3c5a: Verifying Checksum
511136ea3c5a: Download complete
00a0c78eeb6d: Verifying Checksum
00a0c78eeb6d: Download complete
511136ea3c5a: Pull complete
00a0c78eeb6d: Pull complete
834629358fe2: Verifying Checksum
834629358fe2: Download complete
834629358fe2: Pull complete
fedora:21: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Status: Downloaded newer image for fedora:21
time="2015-03-08T16:38:48-04:00" level="info" msg="+job log(pull, fedora:21, )" 
time="2015-03-08T16:38:48-04:00" level="info" msg="-job log(pull, fedora:21, ) = OK (0)" 
time="2015-03-08T16:38:48-04:00" level="info" msg="-job pull(fedora, 21) = OK (0)" 
/bin/sh: line 7: killall: command not found
virt-builder: error: 
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
      killall docker
    : command exited with an error

If reporting bugs, run virt-builder with debugging enabled and include the 
complete output:

  virt-builder -v -x [...]

$

doh :P this is why hacking on these things takes so long... it did seem to work though. Once more with feeling (and killall):

$ cat vb.sh 
#!/bin/bash

virt-builder fedora-21 \
    --install docker,psmisc \
    --run-command '
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
      killall docker
    '

$ ./vb.sh 
[   2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   2.0] Planning how to build this image
[   2.0] Uncompressing
[  14.0] Opening the new disk
[  26.0] Setting a random seed
[  26.0] Installing packages: docker psmisc
[  60.0] Running: 
      docker -H unix:///var/run/docker.sock -d &
      sleep 5
      docker pull fedora:21
      killall docker
    
[ 316.0] Setting passwords
virt-builder: Setting random password of root to hunter2
[ 316.0] Finishing off
                   Output file: fedora-21.img
                   Output size: 4.0G
                 Output format: raw
            Total usable space: 5.2G
                    Free space: 3.8G (72%)

Which seemed to work :)
So I might have a workaround, but there are definitely hacks involved. I see a few solutions:

1) Hopefully the docker people fix this daemon madness
2) If #1 is really impossible for some reason, perhaps virt-builder grows a --docker native command? The problem with this approach is that it might also want a rocket command, and others too. So perhaps something more general would be useful.

Comment 7 Richard W.M. Jones 2015-03-09 12:27:08 UTC
Yeah I suspect hard-coding docker daemons into virt-builder isn't
going to work upstream.  However suggestions for some kind of
generic --daemon or --service option are welcome ...

In general virt-builder's --run-command isn't that similar to
the guest environment.  Some things may be better done in
--firstboot-command.  See also:
http://libguestfs.org/guestfs.3.html#running-commands

Comment 8 James (purpleidea) 2015-03-24 15:00:39 UTC
I've patched vagrant-builder to use this workaround:

https://github.com/purpleidea/vagrant-builder/tree/feat/new-docker

Two notes:

1) I added the --selinux-enabled flag
2) I've wholly not tested this at all (not even once)

Figured I'd put this out here early. More permanent fixes either here and/or in docker are welcome, but probably not urgent at this time.

Cheers,
James

Comment 9 James (purpleidea) 2015-03-30 17:10:22 UTC
FWIW, I've pushed this to git master, and it seems to work :)

w00t! Thanks Rich.

I'd leave this open because it's kind of a hack, but if you prefer we close, that's fine too.

Cheers!


Note You need to log in before you can comment on or make changes to this bug.