Description of problem: You can't 'docker pull' in a virt-builder environment. I suspect this is because something needed by the docker daemon is missing in the virt-builder environment. As an aside, I think it's ridiculous that you need a daemon running to do a docker pull, but fixing that is out of scope for this bug report :( Version-Release number of selected component (if applicable): $ virt-builder --version virt-builder 1.28.6 (on Fedora 21) How reproducible: 100% Steps to Reproduce: $ virt-builder fedora-21 --install docker --run-command 'docker pull fedora:21' [ 8.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz [ 8.0] Planning how to build this image [ 8.0] Uncompressing [ 25.0] Opening the new disk [ 60.0] Setting a random seed [ 60.0] Installing packages: docker [ 154.0] Running: docker pull fedora:21 ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, '') Trying other mirror. Resolving Dependencies --> Running transaction check ---> Package docker-io.x86_64 0:1.5.0-1.fc21 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: docker-io x86_64 1.5.0-1.fc21 updates 5.8 M Transaction Summary ================================================================================ Install 1 Package Total download size: 5.8 M Installed size: 26 M Downloading packages: warning: /var/cache/yum/x86_64/21/updates/packages/docker-io-1.5.0-1.fc21.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY Public key for docker-io-1.5.0-1.fc21.x86_64.rpm is not installed Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64 Importing GPG key 0x95A43F54: Userid : "Fedora (21) <fedora>" Fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54 Package : fedora-repos-21-2.noarch (@anaconda) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : docker-io-1.5.0-1.fc21.x86_64 1/1 Verifying : docker-io-1.5.0-1.fc21.x86_64 1/1 Installed: docker-io.x86_64 0:1.5.0-1.fc21 Complete! time="2015-02-24T13:49:03-05:00" level="fatal" msg="Post http:///var/run/docker.sock/v1.17/images/create?fromImage=fedora%3A21: dial unix /var/run/docker.sock: no such file or directory" virt-builder: error: docker pull fedora:21: command exited with an error If reporting bugs, run virt-builder with debugging enabled and include the complete output: virt-builder -v -x [...] Actual results: fail. Expected results: win. Additional info: I didn't attach the long debug output because this is an easy reproducer. If you can't reproduce, let me know, and I'll attach. Cheers!
The following *nearly* worked for me (and yes, the 'sleep' command *is* necessary, grumble grumble): virt-builder fedora-21 \ --install docker \ --run-command ' docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 ' It eventually prints: [...] 511136ea3c5a: Pulling fs layer 00a0c78eeb6d: Pulling fs layer 834629358fe2: Pulling fs layer 511136ea3c5a: Verifying Checksum 511136ea3c5a: Download complete 00a0c78eeb6d: Verifying Checksum 00a0c78eeb6d: Download complete 511136ea3c5a: Pull complete 00a0c78eeb6d: Pull complete followed by an error, presumably because 'docker pull' exits with the wrong error code, or doesn't print an error message. You could add ||: after the docker pull command to ignore that. Proxy settings from outside virt-builder are, however, not passed through to docker, although as far as I can see that is a docker problem, not a virt-builder problem. Similar to: https://github.com/tmatilai/vagrant-proxyconf/issues/97
Hrm. I ran that and here is what I got: $ virt-builder fedora-21 \ > --install docker \ > --run-command ' > docker -H unix:///var/run/docker.sock -d & > sleep 5 > docker pull fedora:21 > ' [ 4.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz [ 5.0] Planning how to build this image [ 5.0] Uncompressing [ 17.0] Opening the new disk [ 40.0] Setting a random seed [ 40.0] Installing packages: docker [ 102.0] Running: docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 [ 621.0] Setting passwords virt-builder: Setting random password of root to 7II5KPoA7uHByrt1 [ 622.0] Finishing off virt-builder: error: libguestfs error: umount_all: umount: /sysroot: umount: /sysroot: target is busy (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1).) If reporting bugs, run virt-builder with debugging enabled and include the complete output: virt-builder -v -x [...] I didn't dig too deeply, but is this sysroot error caused by something that the daemon is still holding on to? What about the sleep 5, why is this necessary? Is there a clean solution, or a bug I can file with the docker people? Cheers
Yup I suspect the docker daemon is holding /sysroot open. Probably adding `killall docker' as the last command could fix that. Or sending a "terminate the daemon" command if there is one. The sleep 5 was necessary for me - it could be that docker doesn't create the socket before forking into the background (which would be a bug in docker if true).
I've created a docker issue to see if those folks can help suggest a solution or a fix, so that this works together more elegantly. https://github.com/docker/docker/issues/11241
$ cat vb.sh #!/bin/bash virt-builder fedora-21 \ --install docker \ --run-command ' docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 killall docker ' $ ./vb.sh [ 2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz [ 2.0] Planning how to build this image [ 2.0] Uncompressing [ 14.0] Opening the new disk [ 29.0] Setting a random seed [ 29.0] Installing packages: docker [ 62.0] Running: docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 killall docker ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on ftp://ftp.cogeco.net/fedora/linux/releases/21/Everything/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, '') Trying other mirror. Resolving Dependencies --> Running transaction check ---> Package docker-io.x86_64 0:1.5.0-1.fc21 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: docker-io x86_64 1.5.0-1.fc21 updates 5.8 M Transaction Summary ================================================================================ Install 1 Package Total download size: 5.8 M Installed size: 26 M Downloading packages: warning: /var/cache/yum/x86_64/21/updates/packages/docker-io-1.5.0-1.fc21.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY Public key for docker-io-1.5.0-1.fc21.x86_64.rpm is not installed Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64 Importing GPG key 0x95A43F54: Userid : "Fedora (21) <fedora>" Fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54 Package : fedora-repos-21-2.noarch (@anaconda) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : docker-io-1.5.0-1.fc21.x86_64 1/1 Verifying : docker-io-1.5.0-1.fc21.x86_64 1/1 Installed: docker-io.x86_64 0:1.5.0-1.fc21 Complete! time="2015-03-08T16:34:31-04:00" level="info" msg="+job serveapi(unix:///var/run/docker.sock)" time="2015-03-08T16:34:31-04:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)" time="2015-03-08T16:34:31-04:00" level="warning" msg="WARNING: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors" time="2015-03-08T16:34:32-04:00" level="info" msg="+job init_networkdriver()" time="2015-03-08T16:34:32-04:00" level="info" msg="-job init_networkdriver() = OK (0)" time="2015-03-08T16:34:32-04:00" level="info" msg="WARNING: mountpoint for memory not found\n" time="2015-03-08T16:34:32-04:00" level="info" msg="Loading containers: start." time="2015-03-08T16:34:32-04:00" level="info" msg="Loading containers: done." time="2015-03-08T16:34:32-04:00" level="info" msg="docker daemon: 1.5.0 a8a31ef/1.5.0; execdriver: native-0.2; graphdriver: devicemapper" time="2015-03-08T16:34:32-04:00" level="info" msg="+job acceptconnections()" time="2015-03-08T16:34:32-04:00" level="info" msg="-job acceptconnections() = OK (0)" time="2015-03-08T16:34:36-04:00" level="info" msg="POST /v1.17/images/create?fromImage=fedora%3A21" time="2015-03-08T16:34:36-04:00" level="info" msg="+job pull(fedora, 21)" time="2015-03-08T16:34:36-04:00" level="info" msg="+job resolve_repository(fedora)" time="2015-03-08T16:34:36-04:00" level="info" msg="-job resolve_repository(fedora) = OK (0)" time="2015-03-08T16:34:36-04:00" level="info" msg="+job trust_update_base()" time="2015-03-08T16:34:36-04:00" level="info" msg="-job trust_update_base() = OK (0)" time="2015-03-08T16:34:37-04:00" level="info" msg="+job trust_key_check(/library/fedora)" time="2015-03-08T16:34:37-04:00" level="info" msg="-job trust_key_check(/library/fedora) = OK (0)" time="2015-03-08T16:34:37-04:00" level="info" msg="Image manifest for fedora:21 has been verified" 511136ea3c5a: Pulling fs layer 00a0c78eeb6d: Pulling fs layer 834629358fe2: Pulling fs layer 511136ea3c5a: Verifying Checksum 511136ea3c5a: Download complete 00a0c78eeb6d: Verifying Checksum 00a0c78eeb6d: Download complete 511136ea3c5a: Pull complete 00a0c78eeb6d: Pull complete 834629358fe2: Verifying Checksum 834629358fe2: Download complete 834629358fe2: Pull complete fedora:21: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Status: Downloaded newer image for fedora:21 time="2015-03-08T16:38:48-04:00" level="info" msg="+job log(pull, fedora:21, )" time="2015-03-08T16:38:48-04:00" level="info" msg="-job log(pull, fedora:21, ) = OK (0)" time="2015-03-08T16:38:48-04:00" level="info" msg="-job pull(fedora, 21) = OK (0)" /bin/sh: line 7: killall: command not found virt-builder: error: docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 killall docker : command exited with an error If reporting bugs, run virt-builder with debugging enabled and include the complete output: virt-builder -v -x [...] $ doh :P this is why hacking on these things takes so long... it did seem to work though. Once more with feeling (and killall): $ cat vb.sh #!/bin/bash virt-builder fedora-21 \ --install docker,psmisc \ --run-command ' docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 killall docker ' $ ./vb.sh [ 2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz [ 2.0] Planning how to build this image [ 2.0] Uncompressing [ 14.0] Opening the new disk [ 26.0] Setting a random seed [ 26.0] Installing packages: docker psmisc [ 60.0] Running: docker -H unix:///var/run/docker.sock -d & sleep 5 docker pull fedora:21 killall docker [ 316.0] Setting passwords virt-builder: Setting random password of root to hunter2 [ 316.0] Finishing off Output file: fedora-21.img Output size: 4.0G Output format: raw Total usable space: 5.2G Free space: 3.8G (72%) Which seemed to work :) So I might have a workaround, but there are definitely hacks involved. I see a few solutions: 1) Hopefully the docker people fix this daemon madness 2) If #1 is really impossible for some reason, perhaps virt-builder grows a --docker native command? The problem with this approach is that it might also want a rocket command, and others too. So perhaps something more general would be useful.
Yeah I suspect hard-coding docker daemons into virt-builder isn't going to work upstream. However suggestions for some kind of generic --daemon or --service option are welcome ... In general virt-builder's --run-command isn't that similar to the guest environment. Some things may be better done in --firstboot-command. See also: http://libguestfs.org/guestfs.3.html#running-commands
I've patched vagrant-builder to use this workaround: https://github.com/purpleidea/vagrant-builder/tree/feat/new-docker Two notes: 1) I added the --selinux-enabled flag 2) I've wholly not tested this at all (not even once) Figured I'd put this out here early. More permanent fixes either here and/or in docker are welcome, but probably not urgent at this time. Cheers, James
FWIW, I've pushed this to git master, and it seems to work :) w00t! Thanks Rich. I'd leave this open because it's kind of a hack, but if you prefer we close, that's fine too. Cheers!