Bug 1042786 - Docker can't talk to the API as certificate can't be verfified
Summary: Docker can't talk to the API as certificate can't be verfified
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: docker-io
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-13 11:53 UTC by Peter Meier
Modified: 2014-07-01 22:59 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-15 20:42:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Peter Meier 2013-12-13 11:53:06 UTC
Description of problem:

I can't use docker, as it can't talk to the API as api.go is not able to verifiy the certificate of docker's API.

Version-Release number of selected component (if applicable):

# rpm -qi docker-io
Name        : docker-io
Version     : 0.7.0
Release     : 14.fc19
Architecture: x86_64
Install Date: Tue 10 Dec 2013 07:12:16 PM CET
Group       : Unspecified
Size        : 12003115
License     : ASL 2.0
Signature   : RSA/SHA256, Tue 03 Dec 2013 01:17:40 AM CET, Key ID 07477e65fb4b18e6
Source RPM  : docker-io-0.7.0-14.fc19.src.rpm
Build Date  : Mon 02 Dec 2013 05:06:54 PM CET
Build Host  : buildvm-12.phx2.fedoraproject.org


How reproducible:

Install docker-io, try to run a search -> fail


Steps to Reproduce:
1. yum install docker-io
2. systemctl start docker
3. docker search fedora

Actual results:

$ docker search fedora 
2013/12/13 12:41:35 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority

/var/log/messages
Dec 13 12:51:10 foo docker[14359]: 2013/12/13 12:51:10 GET /v1.7/images/search?term=fedora
Dec 13 12:51:10 foo docker[14359]: [error] api.go:1034 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority
Dec 13 12:51:10 foo docker[14359]: [error] api.go:82 HTTP Error: statusCode=500 Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority


Expected results:

Givme the fedora images


Additional info:

The is no problem to query this URL from curl nor from wget:

# curl https://index.docker.io/v1/search?q=fedora
{"query": "fedora", "num_results": 11, "results": [{"name": "mattdm/fedora", "description": "A basic Fedora image corresponding roughly to a minimal install, minus some things which don't make sense in a container. Use tag `f19` for Fedora 19."}, {"name": "alexl/fedora-19", "description": "Minimal base images based on Fedora 19"}, {"name": "simoncadman/fedora-20", "description": "Updated fedora 20, based on goldmann/f20 , includes packages for building rpms"}, {"name": "mattdm/fedora-small", "description": "A small Fedora image on which to build. Contains just enough that you'll be able to run `yum install` in your dockerfiles to create something useful. Use tag `f19` for Fedora 19."}, {"name": "philips/fedora", "description": ""}, {"name": "kraman/fedora_cfn", "description": "mattdm/fedora:f19 + cloud-Init\n(SSH key setting has been disabled)"}, {"name": "dgarcia/fedora18base", "description": ""}, {"name": "goldmann/f20", "description": "Fedora 20 repository"}, {"name": "philips/riak-base", "description": "Base Fedora box with Riak installed."}, {"name": "jumanjiman/eiffelstudio", "description": "EiffelStudio IDE in a Docker container (fedora at the moment)"}, {"name": "svendowideit/publican", "description": "Publish DocBook documentation using publican (running in a fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican build"}]}


# wget -O /dev/stdout https://index.docker.io/v1/search?q=fedora
--2013-12-13 12:52:21--  https://index.docker.io/v1/search?q=fedora
Resolving index.docker.io (index.docker.io)... 54.224.119.89, 54.234.135.251
Connecting to index.docker.io (index.docker.io)|54.224.119.89|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘/dev/stdout’

    [<=>                                                                                             ] 0           --.-K/s              {"query": "fedora", "num_results": 11, "results": [{"name": "mattdm/fedora", "description": "A basic Fedora image corresponding roughly to a minimal install, minus some things which don't make sense in a container. Use tag `f19` for Fedora 19."}, {"name": "alexl/fedora-19", "description": "Minimal base images based on Fedora 19"}, {"name": "simoncadman/fedora-20", "description": "Updated fedora 20, based on goldmann/f20 , includes packages for building rpms"}, {"name": "mattdm/fedora-small", "description": "A small Fedora image on which to build. Contains just enough that you'll be able to run `yum install` in your dockerfiles to create something useful. Use tag `f19` for Fedora 19."}, {"name": "philips/fedora", "description": ""}, {"name": "kraman/fedora_cfn", "description": "mattdm/fedora:f19 + cloud-Init\n(SSH key setting has been disabled)"}, {"name": "dgarcia/fedora18base", "description": ""}, {"name": "goldmann/f20", "description": "Fedora 20 repository"}, {"name": "philips/riak-base", "description": "Base Fedora box with Riak installed."}, {"name": "jumanjiman/eiffelstudio", "description": "EiffelStudio IDE in a Docker container (fedora at the moment)"}, {"name": "svendowideit/publican", "description": "Publish DocBook documentation using publican (running in a fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican build"}    [ <=>                                                                                            ] 1,373       --.-K/s   in 0.001s  

2013-12-13 12:52:21 (1.24 MB/s) - ‘/dev/stdout’ saved [1373]

Comment 1 Lokesh Mandvekar 2013-12-14 23:35:37 UTC
Hi Peter, 

Can you check if this occurs with 0.7.1-1 (currently in testing repo)? If yes, can you also check with upstream released binary? http://docs.docker.io/en/latest/installation/binaries/

I'm seeing something similar to this (Bug 1041400) on rawhide with 0.7.1-1 and also with the upstream binary.

Comment 2 Peter Meier 2013-12-16 22:34:39 UTC
The certificate verification works with 0.7.1-1 (in updates-testing).

However docker seems now to use another api-endpoint which does not work:

$ docker -v
Docker version 0.7.1, build e39d35d/0.7.1
$ rpm -qa | grep docker-io
docker-io-0.7.1-1.fc19.x86_64

$ docker search fedora
2013/12/16 23:31:02 Error: Not Found

Dec 16 23:27:04 foo docker[14359]: 2013/12/16 23:27:04 GET /v1.8/images/search?term=fedora

And indeed this api calls gives a 404:


$ curl -I https://index.docker.io/v1.8/search?q=fedora
HTTP/1.1 404 Not Found
server: nginx/1.2.1
date: Mon, 16 Dec 2013 22:31:47 GMT
content-type: text/html; charset=utf-8
connection: close
vary: Cookie

Looks like a different problem, so closing this bug.

Comment 3 Peter Meier 2013-12-16 22:44:47 UTC
Interesting due to #1038329 I restarted the daemon and now it fails again:

Dec 16 23:31:02 foo docker[14359]: 2013/12/16 23:31:02 GET /v1.8/images/search?term=fedora
Dec 16 23:40:29 foo docker[14359]: 2013/12/16 23:40:29 Received signal 'terminated', exiting
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] +job initapi()
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()] Creating server
Dec 16 23:40:29 foo docker[23426]: Loading containers:  #010done.
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()] Creating pidfile
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()] Setting up signal traps
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] Register(create) (handlers=map[initapi:0x496300])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] Register(start) (handlers=map[initapi:0x496300 create:0x4b4160])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] Register(serveapi) (handlers=map[initapi:0x496300 create:0x4b4160 start:0x4b41a0])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] -job initapi() = OK (0)
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] +job serveapi(unix:///var/run/docker.sock) 
Dec 16 23:40:29 foo docker[23426]: 2013/12/16 23:40:29 Listening for HTTP on /var/run/docker.sock (unix) 
Dec 16 23:40:54 foo docker[23426]: 2013/12/16 23:40:54 GET /v1.8/images/search?term=fedora
Dec 16 23:40:54 foo docker[23426]: [error] api.go:1065 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority 
Dec 16 23:40:54 foo docker[23426]: [error] api.go:87 HTTP Error: statusCode=500 Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority
Dec 16 23:41:15 foo docker[23426]: 2013/12/16 23:41:15 GET /v1.8/images/search?term=fedora
Dec 16 23:41:16 foo docker[23426]: [error] api.go:1065 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority 
Dec 16 23:41:16 foo docker[23426]: [error] api.go:87 HTTP Error: statusCode=500 Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority


So still no luck :(

The interesting part is that it looks like it is first trying v1.8 and then then v1 API and only barfs on the last one.

Comment 4 Lokesh Mandvekar 2013-12-19 19:35:00 UTC
Peter, could you check with 0.7.2 ?

this should be going into updates-testing repo for f19 and f20 soon. get the f19 rpm from here: http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64/docker-io-0.7.2-1.fc19.x86_64.rpm

Comment 5 Peter Meier 2013-12-19 20:16:11 UTC
Unfortunately not:

$ rpm -Uvh http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64/docker-io-0.7.2-1.fc19.x86_64.rpm
Retrieving http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64/docker-io-0.7.2-1.fc19.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-io-0.7.2-1.fc19           ################################# [ 50%]
Cleaning up / removing...
   2:docker-io-0.7.1-1.fc19           ################################# [100%]

$ service docker restart
Redirecting to /bin/systemctl restart  docker.service

$ docker search fedora
2013/12/19 21:05:32 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority

$ docker -v
Docker version 0.7.2, build 28b162e/0.7.2

$ rpm -qi docker-io
Name        : docker-io
Version     : 0.7.2
Release     : 1.fc19
Architecture: x86_64
Install Date: Thu 19 Dec 2013 09:05:18 PM CET
Group       : Unspecified
Size        : 12776659
License     : ASL 2.0
Signature   : (none)
Source RPM  : docker-io-0.7.2-1.fc19.src.rpm
Build Date  : Wed 18 Dec 2013 08:09:48 PM CET
Build Host  : buildvm-24.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.docker.io
Summary     : Automates deployment of containerized applications
Description :
Docker is an open-source engine that automates the deployment of any
application as a lightweight, portable, self-sufficient container that will
run virtually anywhere.

Docker containers can encapsulate any payload, and will run consistently on
and between virtually any server. The same container that a developer builds
and tests on a laptop will run at scale, in production*, on VMs, bare-metal
servers, OpenStack clusters, public instances, or combinations of the above.


$ curl https://index.docker.io/v1/search?q=fedora 
{"query": "fedora", "num_results": 14, "results": [{"name": "mattdm/fedora", "description": "A basic Fedora image corresponding roughly to a minimal install, minus some things which don't make sense in a container. Use tag `f20` for Fedora 20 or `f19` for Fedora 19."}, {"name": "alexl/fedora-19", "description": "Minimal base images based on Fedora 19"}, {"name": "simoncadman/fedora-20", "description": "Updated fedora 20, based on goldmann/f20 , includes packages for building rpms"}, {"name": "mattdm/fedora-small", "description": "A small Fedora image on which to build. Contains just enough that you'll be able to run `yum install` in your dockerfiles to create something useful. Use tag `f19` for Fedora 19."}, {"name": "philips/fedora", "description": ""}, {"name": "lsm5/fedora-imagebuilder", "description": ""}, {"name": "lzap/fedora-foreman-git-base", "description": ""}, {"name": "lzap/fedora-foreman-git-stable", "description": "Foreman stable installation from Git\n\nhttps://github.com/lzap/foreman-docker"}, {"name": "kraman/fedora_cfn", "description": "mattdm/fedora:f19 + cloud-Init\n(SSH key setting has been disabled)"}, {"name": "dgarcia/fedora18base", "description": ""}, {"name": "goldmann/f20", "description": "Fedora 20 repository"}, {"name": "philips/riak-base", "description": "Base Fedora box with Riak installed."}, {"name": "jumanjiman/eiffelstudio", "description": "EiffelStudio IDE in a Docker container (fedora at the moment)"}, {"name": "svendowideit/publican", "description": "Publish DocBook documentation using publican (running in a fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican build"}]}


$ grep docker /var/log/messages | tail
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(tag) (handlers=map[export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00 initapi:0x496520 version:0x4b1720 create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(resize) (handlers=map[initapi:0x496520 version:0x4b1720 create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80 export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(commit) (handlers=map[export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00 resize:0x4b6bc0 initapi:0x496520 version:0x4b1720 create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(info) (handlers=map[initapi:0x496520 version:0x4b1720 create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80 commit:0x4b6c00 export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00 resize:0x4b6bc0])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] -job initapi() = OK (0)
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] +job serveapi(unix:///var/run/docker.sock)
Dec 19 21:05:24 gasteiz docker[7093]: 2013/12/19 21:05:24 Listening for HTTP on /var/run/docker.sock (unix)
Dec 19 21:05:32 gasteiz docker[7093]: 2013/12/19 21:05:32 GET /v1.8/images/search?term=fedora
Dec 19 21:05:32 gasteiz docker[7093]: [error] api.go:1062 Error: Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority
Dec 19 21:05:32 gasteiz docker[7093]: [error] api.go:87 HTTP Error: statusCode=500 Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown authority

I don't understand why it fails with docker, while curl works...

Comment 6 Lokesh Mandvekar 2013-12-20 01:33:05 UTC
hmm, I can't seem to replicate it, perhaps you might wanna check if this helps: https://groups.google.com/d/msg/golang-nuts/vWewH0Wum90/4A4SNmdlb8gJ

Comment 7 Peter Meier 2014-01-15 20:42:55 UTC
Finally, I found the solution. And it's totally my own fault.

From previous tinkering with extending the certificate chain I had a link /etc/pki/tls/certs/ca-certificates.crt pointing to /etc/pki/tls/certs/ca-bundle.trust.crt which caused these failures.

Removing that faulty link fixed the isuee.


Note You need to log in before you can comment on or make changes to this bug.