Bug 1026976

Summary: "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image
Product: [Fedora] Fedora Reporter: Seth Jennings <sjenning>
Component: docker-ioAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: alexl, bkozdemb, dkl, golang-updates, jkeck, lsm5, mattdm, mgoldman, vbatts
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-03 13:17:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Seth Jennings 2013-11-05 18:49:13 UTC 
Description of problem:

Running container fails with:

Error: Error starting container 43e36bc82ecd: Error adding device for '0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651': can't find device for parent ''

Additionally /var/log/messages has this:
Nov  5 10:20:20 docker kernel: [   85.389079] device-mapper: space map common: index_check failed: blocknr 0 != wanted 314
Nov  5 10:20:20 docker kernel: [   85.389102] device-mapper: block manager: index validator check failed for block 314
Nov  5 10:20:20 docker kernel: [   85.389284] device-mapper: transaction manager: couldn't open metadata space map
Nov  5 10:20:20 docker kernel: [   85.389295] device-mapper: thin metadata: tm_open_with_sm failed
Nov  5 10:20:20 docker kernel: [   85.390964] device-mapper: table: 253:1: thin-pool: Error creating metadata object
Nov  5 10:20:20 docker kernel: [   85.391095] device-mapper: ioctl: error adding target to table

Version-Release number of selected component (if applicable):
# rpm -qa docker-io
docker-io-0.7-0.13.dm.fc19.x86_64

# uname -a
Linux ... 3.11.6-200.fc19.x86_64 #1 SMP Fri Oct 18 22:34:18 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:
Every time

Steps to Reproduce:
1. Install docker-io-0.7-0.13.dm.fc19.x86_64.rpm
2. Start docker daemon (docker -d &)
3. Download mattdm/fedora images (docker pull mattdm/fedora)
4. Try to start a fedora image (docker run -i -t mattdm/fedora /bin/bash)

Actual results:
Error: Error starting container 96c6d2706331: Error adding device for '0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651': can't find device for parent ''

Expected results:
Shell access to fedora container

Additional info:

My rootfs is on an LV fedora-root in VG fedora

# dmsetup ls --tree
fedora-root (253:0)
 └─ (252:2)

Not sure if that would make a difference.
Comment 1 Alexander Larsson 2013-11-06 09:30:01 UTC 
The device mapper errors seem very bad, I have no idea what exactly goes wrong, but it seems like the devicemapper block devices are bad and it just fails to set up the devmapper devices, and then later fails due to them not existing.

Its a bit hard to debug this with the upstream dm work in flux though.
Comment 2 Lokesh Mandvekar 2013-11-24 00:33:44 UTC 
Seth, could you check if this issue persists with the latest rpm?
Comment 3 Seth Jennings 2013-11-25 17:12:51 UTC 
I don't observe this with docker-io-0.7-0.20.rc7.fc19.x86_64:

[root@localhost ~]# docker -d &
[1] 915
[root@localhost ~]# [/var/lib/docker|72633d2e] +job initapi()
[/var/lib/docker|72633d2e.initapi()] Creating server
Loading containers: done.
[/var/lib/docker|72633d2e.initapi()] Creating pidfile
[/var/lib/docker|72633d2e.initapi()] Setting up signal traps
[/var/lib/docker|72633d2e] Register(create) (handlers=map[initapi:0x491200])
[/var/lib/docker|72633d2e] Register(start) (handlers=map[initapi:0x491200 create:0x4aec70])
[/var/lib/docker|72633d2e] Register(serveapi) (handlers=map[initapi:0x491200 create:0x4aec70 start:0x4aecd0])
[/var/lib/docker|72633d2e] -job initapi() = OK (0)
[/var/lib/docker|72633d2e] +job serveapi(unix:///var/run/docker.sock)
2013/11/25 11:02:17 Listening for HTTP on /var/run/docker.sock (unix)

[root@localhost ~]# rpm -qa docker-io
docker-io-0.7-0.20.rc7.fc19.x86_64

[root@localhost ~]# docker version
Client version: 0.7
Go version (client): go1.1.2
Git commit (client): ea7811c/0.20.rc7.fc19
2013/11/25 11:08:02 GET /v1.7/version
Server version: 0.7
Git commit (server): ea7811c/0.20.rc7.fc19
Go version (server): go1.1.2
Last stable version: 0.6.7, please update docker

[root@localhost ~]# docker images
2013/11/25 11:08:09 GET /v1.7/images/json
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
mattdm/fedora       f20                 1bdd590de956        4 weeks ago         348 MB (virtual 348 MB)
mattdm/fedora       latest              0f3e92b4e94d        4 weeks ago         348 MB (virtual 348 MB)
mattdm/fedora       f19                 50f374c05c2c        9 weeks ago         349 MB (virtual 349 MB)
mattdm/fedora       f20rc3.small        97fc5bf7f8d4        9 weeks ago         198.4 MB (virtual 198.4 MB)

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.11.9-200.fc19.x86_64 #1 SMP Wed Nov 20 21:22:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# docker run -i -t mattdm/fedora /bin/bash
2013/11/25 11:08:23 POST /v1.7/containers/create
[/var/lib/docker|72633d2e] +job create()
e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610
[/var/lib/docker|72633d2e] -job create() = OK (0)
2013/11/25 11:08:23 POST /v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/attach?stderr=1&stdin=1&stdout=1&stream=1
2013/11/25 11:08:23 POST /v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/start
                                                                                                                [/var/lib/docker|72633d2e] +job start(e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610)
                                                              bash-4.2# [/var/lib/docker|72633d2e] -job start(e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610) = OK (0)
                               2013/11/25 11:08:23 POST /v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/resize?h=37&w=153
bash-4.2# exit
exit
2013/11/25 11:08:27 GET /v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/json

[root@localhost ~]# dmsetup ls --tree
docker-252:2-307531-1bdd590de956281c7a5b1394cc0801ee879bc220be52e399eb61a9c3f2810c0d (253:2)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-base (253:1)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610-init (253:8)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-7c46893c7790547d9d6e604cc327f246fd670c872378a6a203e1bf7ae4204c3c (253:7)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651 (253:5)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-7c46893c7790547d9d6e604cc327f246fd670c872378a6a203e1bf7ae4204c3c-init (253:6)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-50f374c05c2c07d28620824a6f8427d50879d92f5be6b318d3340f4f0f6120a2 (253:4)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610 (253:9)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-97fc5bf7f8d42606fa896e1d391a0b882f78322ce0ff77c03fbd7f8e3b7a73ed (253:3)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
Comment 4 Marek Goldmann 2013-12-03 13:17:07 UTC 
I'll close this issue, feel free to reopen it, but I'm pretty sure it was fixed with the 0.7.0 release - I don't see this issue.