| Summary: | docker-btrfs: error locating sandbox id | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | michae1t |
| Component: | docker | Assignee: | Antonio Murdaca <amurdaca> |
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 24 | CC: | adimania, admiller, amurdaca, dwalsh, ichavero, jcajka, jchaloup, lsm5, marianne, michae1t, miminar, nalin, vbatts, vbellur |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-08 14:15:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
michae1t
2016-04-18 20:15:01 UTC
/var/log/audit/audit.log clear Looks like one of our patches is causing this. Could you provide both the systemd unit file and /etc/sysconfig/docker (also before you reinstalled the upstream binary) Could you also test our rpm with another storage driver? like devicemapper or overlay. Apart from the daemon logs nothing clearly shows that the error could be related to the storage driver, instead, I think the "System error" comes from libcontainer and could be something cgroups related (we have some patches that play with cgroups in 1.10.3 that upstream isn't carrying) I've just tried on a vm with /var/lib/docker on a loopback device backed by btrfs on f24 - everything works fine on a fresh installation. Are you upgrading docker version? I suspect something in the unit file could also cause this. This is a clean install. No upgrade, no modifications to unit files or other config. For the debug above I started "docker daemon" on the command line without systemd. I do not have experence with config of the docker server. If you can provide instruction on how to substitute the storage driver and your preferred one I can run a test. I link to a decent existing tutorial will also do. Either way, /etc/sysconfig/docker is empty. As for the rest: > cat /lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target Wants=docker-storage-setup.service [Service] Type=notify EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network Environment=GOTRACEBACK=crash ExecStart=/usr/bin/docker daemon \ --exec-opt native.cgroupdriver=systemd \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $INSECURE_REGISTRY LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal [Install] WantedBy=multi-user.target > cat /lib/systemd/system/docker-storage-setup.service [Unit] Description=Docker Storage Setup After=cloud-final.service Before=docker.service [Service] Type=oneshot ExecStart=/usr/bin/docker-storage-setup EnvironmentFile=-/etc/sysconfig/docker-storage-setup [Install] WantedBy=multi-user.target [root@pixel Downloads]# > cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS='--selinux-enabled --log-driver=journald' DOCKER_CERT_PATH=/etc/docker # Enable insecure registry communication by appending the registry URL # to the INSECURE_REGISTRY variable below and uncommenting it # INSECURE_REGISTRY='--insecure-registry ' # On SELinux System, if you remove the --selinux-enabled option, you # also need to turn on the docker_transition_unconfined boolean. # setsebool -P docker_transition_unconfined # Location used for temporary files, such as those created by # docker load and build operations. Default is /var/lib/docker/tmp # Can be overriden by setting the following environment variable. # DOCKER_TMPDIR=/var/tmp # Controls the /etc/cron.daily/docker-logrotate cron job status. # To disable, uncomment the line below. # LOGROTATE=false > cat /etc/sysconfig/docker-storage # This file may be automatically generated by an installation program. # By default, Docker uses a loopback-mounted sparse file in # /var/lib/docker. The loopback makes it slower, and there are some # restrictive defaults, such as 100GB max storage. # If your installation did not set a custom storage for Docker, you # may do it below. # Example: Use a custom pair of raw logical volumes (one for metadata, # one for data). # DOCKER_STORAGE_OPTIONS = --storage-opt dm.metadatadev=/dev/mylogvol/my-docker-metadata --storage-opt dm.datadev=/dev/mylogvol/my-docker-data DOCKER_STORAGE_OPTIONS= > cat /etc/sysconfig/docker-storage-setup # Edit this file to override any configuration options specified in # /usr/lib/docker-storage-setup/docker-storage-setup. # # For more details refer to "man docker-storage-setup" I cannot reproduce your issue on a fresh vm with /var/lib/docker on a btrfs volume. It's weird upstream docker worked fine. Where are you running docker? Physical? Virtual machine? Could you try the whole thing out again on a fresh machine (even virtual)? I assume your /var/lib/docker is on a btrfs volume. If you could just redo everything on a fresh machine and just ststemctl enable docker, docker run Fedora that would help. Otherwise we'll configure the daemon to use another storage driver (if the issue persists). This is a physical machine. Blowing away the system is not practical right now. I suspect I will get the same result as you with the virtual machine. The setup storage setup is atypical using two btrfs filesystems on two devices. Both are encrypted with luks. You can see the config in my initial comment. It seems unlikely this would cause such an issue in general but if you are not able to repro with a more straightforward setup it might be worth mixing things up. There must be a genuine issue among the patches, be it small and be it particular to my system. I am happy to try another storage driver or bisect these patches mentioned if you can provide the RPMS or an easy way to produce them. The other option if you don't think this is a big deal we can let it sit and see if it comes up as more people use docker. By the sounds of it is also possible that this issue has existed through previous releases, my previous install had a more basic filesystem layout. This is not a huge deal to me as mainstream works fine and is running a newer version. I can work off that. ===== Here is the output of info on mainline servers in case there is any applicable difference. The btrfs storage driver looks newer than the same docker-engine version in the fedora package. > docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 6 Server Version: 1.10.3 Storage Driver: btrfs Build Version: Btrfs v4.3.1 Library Version: 101 Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 4.5.1-300.fc24.x86_64 Operating System: Fedora 24 (Twenty Four) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.61 GiB Name: pixel.home.local ID: EMW2:BU6Y:FSIG:AKA2:HUEZ:IB4V:NDZ2:V4G3:RRWQ:KZBB:A3IV:42BW Registry: https://index.docker.io/v1/ > docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 6 Server Version: 1.11.0 Storage Driver: btrfs Build Version: Btrfs v4.4.1 Library Version: 101 Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge null host Kernel Version: 4.5.1-300.fc24.x86_64 Operating System: Fedora 24 (Twenty Four) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.61 GiB Name: pixel.home.local ID: EMW2:BU6Y:FSIG:AKA2:HUEZ:IB4V:NDZ2:V4G3:RRWQ:KZBB:A3IV:42BW Docker Root Dir: /var/lib/docker Debug mode (client): false Debug mode (server): false Registry: https://index.docker.io/v1/ Another curiosity of this state... I noticed two fedora entries in the images output now with the working mainline package - one that could not be removed without deleting the contents of /var/lib/docker. I was able to repro by deleting the /var/lib/docker installing the fedora package again - which again did not work - and re migrating to mainline. > docker images REPOSITORY TAG IMAGE ID CREATED SIZE fedora latest ddd5c9c1d0f2 6 weeks ago > docker run fedora bash Unable to find image 'fedora:latest' locally latest: Pulling from library/fedora a3ed95caeb02: Already exists 236608c7b546: Already exists Digest: sha256:1fa98be10c550ffabde65246ed2df16be28dc896d6e370dab56b98460bd27823 Status: Downloaded newer image for fedora:latest > docker images REPOSITORY TAG IMAGE ID CREATED SIZE fedora latest ddd5c9c1d0f2 6 weeks ago 204.7 MB fedora latest ddd5c9c1d0f2 6 weeks ago 204.7 MB > docker rmi fedora Untagged: fedora:latest > docker images REPOSITORY TAG IMAGE ID CREATED SIZE fedora latest ddd5c9c1d0f2 6 weeks ago 204.7 MB Okay, this is definitely a general issue and not related to my setup nor the multiple filesystems nor luks. I installed Fedora 24 Alpha net install to VirtualBox container with a /boot ext3 and /home btrfs no luks. Same issue that I am getting on the native machine. Can share the image if you can't repro. It is about 6GB. Lets grab the image. Yes, thanks, I'm going to try our another virtual machine with full btrfs root fs, maybe this is causing it, because I tested with ext4 + just /var/lib/docker on btrfs I have provided a link out of band to the vbox image on s3 to your emails. Have you guys been able to repro? I encounter a similar problem (without btrfs) in Fedora 24. Docker related packages in my Fedora 24: docker-distribution-2.4.1-1.fc24.x86_64 docker-v1.10-migrator-1.10.3-50.gita612434.fc24.x86_64 devassistant-dap-docker-0.11-3.fc24.noarch docker-forward-journald-1.10.3-24.gitf476348.fc23.x86_64 docker-1.10.3-52.git8b7fa4a.fc24.x86_64 docker-selinux-1.10.3-52.git8b7fa4a.fc24.x86_64 Would any logs or further information about this problem help in resolution? Thanks! We don't do a lot of testing with btrfs, most of our efforts are spent on OverlayFS and Devicemapper. (In reply to Daniel Walsh from comment #20) > We don't do a lot of testing with btrfs, most of our efforts are spent on > OverlayFS and Devicemapper. This is without btrfs in the mix: #docker info Containers: 12 Running: 0 Paused: 0 Stopped: 12 Images: 1 Server Version: 1.10.3 Storage Driver: devicemapper This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |