Bug 1326110 - Unable to create containers with Kubernetes master and Docker 1.9.1-9
Summary: Unable to create containers with Kubernetes master and Docker 1.9.1-9
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Antonio Murdaca
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-11 20:53 UTC by Andy Goldstein
Modified: 2016-06-10 02:26 UTC (History)
13 users (show)

Fixed In Version: docker-1.10.3-24.gitf476348.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-10 02:26:58 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Seeing this issue with Atomic App on Fedora as well (3.31 KB, text/plain)
2016-05-25 18:28 UTC, Dusty Mabe
no flags Details

Description Andy Goldstein 2016-04-11 20:53:40 UTC
Description of problem: Kubernetes master (commit 391ff9d) is unable to create containers with docker-1.9.1-9


Version-Release number of selected component (if applicable): docker-1.9.1-9.gitee06d03.fc23.x86_64


How reproducible: 100%


Steps to Reproduce:
1. git clone https://github.com/kubernetes/kubernetes to $GOPATH/src/k8s.io/kubernetes
2. cd $GOPATH/src/k8s.io/kubernetes
3. sudo ./hack/local-up-cluster.sh
4. kubectl create -f docs/user-guide/pod.yaml
5. kubectl describe pod/nginx

Actual results: Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: SHM size must be greater then 0"


Expected results: Pod runs


Additional info:

Kubernetes has switched from go-dockerclient to docker's engine-api. As part of this change, the HostConfig's ShmSize is an int64, which means it gets defaulted to 0. go-dockerclient didn't contain this field, so it wasn't getting set. But now that it is defaulted to 0, when combined with https://github.com/projectatomic/docker/blob/fedora-1.9/daemon/daemon_unix.go#L138, you get the error listed above.

It looks like docker 1.10 shouldn't have this problem: https://github.com/projectatomic/docker/blob/fedora-1.10/daemon/daemon_unix.go#L402.

Would it be possible to either get an updated 1.9 or to have 1.10 in Fedora 23?

Comment 1 Antonio Murdaca 2016-04-11 21:07:20 UTC
The change in our projectatomic/docker is a backport from 1.10 - 1.10 changed a bit from 1.9 and that check has been removed. I'm sure it's an error if you provide a negative shmSize, not sure if upstream is a bug or not. Can you actually create this pod if you use 1.10? shmSize should be validated in both 1.9 and 1.10 when is < 0

Comment 2 Andy Goldstein 2016-04-11 21:10:57 UTC
I will test with 1.10 on rawhide later tonight and report back.

However, this pod creates a container that does not have ShmSize set... It's not that Kube is setting it to 0 and we can just fix Kube. We need to figure out how to make sure this will work in Docker instead of it being a validation error.

Comment 3 Antonio Murdaca 2016-04-11 21:13:11 UTC
Found the patch and the login behind the change in 1.10 - will fix this for 1.9. It would be good if you could still test this in 1.10. Thanks!

Comment 4 Andy Goldstein 2016-04-12 00:38:52 UTC
1.10 on rawhide works fine

Comment 5 Antonio Murdaca 2016-04-12 10:01:43 UTC
https://github.com/projectatomic/docker/pull/98

Comment 6 Antonio Murdaca 2016-04-12 17:40:58 UTC
just tested and seems to be working now:

19:40:29 amurdaca at localhost in src/k8s.io/kubernetes ‹master*› cluster/kubectl.sh describe pod/nginx
Name:		nginx
Namespace:	default
Node:		127.0.0.1/127.0.0.1
Start Time:	Tue, 12 Apr 2016 19:37:18 +0200
Labels:		app=nginx
Status:		Running
IP:		172.17.0.2
Controllers:	<none>
Containers:
  nginx:
    Container ID:	docker://dfea384c58a8ca44e72607cb947bfa181abab47bbeffc368e07fcb85c63d6b28
    Image:		nginx
    Image ID:		docker://e32087da8ee6dfa45221c48670fa9475f3d8a53a0e9ccabef4f741c62c77d49b
    Port:		80/TCP
    QoS Tier:
      cpu:			BestEffort
      memory:			BestEffort
    State:			Running
      Started:			Tue, 12 Apr 2016 19:40:22 +0200
    Ready:			True
    Restart Count:		0
    Environment Variables:	<none>
Conditions:
  Type		Status
  Ready 	True 
Volumes:
  default-token-rye32:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-rye32
Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath		Type		Reason		Message
  ---------	--------	-----	----			-------------		--------	------		-------
  3m		3m		1	{default-scheduler }				Normal		Scheduled	Successfully assigned nginx to 127.0.0.1
  3m		3m		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Pulling		pulling image "nginx"
  9s		9s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Pulled		Successfully pulled image "nginx"
  9s		9s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Created		Created container with docker id dfea384c58a8
  8s		8s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Started		Started container with docker id dfea384c58a8

Comment 7 Antonio Murdaca 2016-04-12 17:44:29 UTC
tested also the old 1.9 which showed me the error you are reporting. So this fix is working fine now.

Comment 8 Daniel Walsh 2016-04-12 17:48:48 UTC
Antonio which version of docker package is this fixed in?

Comment 9 Antonio Murdaca 2016-04-12 17:50:22 UTC
docker-1.9 in rhel7-1.9 branch -> https://github.com/projectatomic/docker/pull/98
I didn't make a PR against fedora-1.9 branch because we don't seem to be shipping fixes for fedora anymore.

Comment 10 Andy Goldstein 2016-04-12 18:54:53 UTC
We need something in Fedora 23 that isn't broken.

Comment 11 Antonio Murdaca 2016-04-12 19:05:45 UTC
confirmed this is working fine with docker-1.10.3

Comment 12 Daniel Walsh 2016-04-12 19:43:43 UTC
docker-1.10.3 is available for f23, I believe.  Just in updates-testing, because we have been asked not to release it.

Comment 13 Andy Goldstein 2016-04-12 19:48:10 UTC
It's not there yet

Comment 14 Daniel Walsh 2016-04-12 20:03:14 UTC
Well it has been pulled back so that we can update the docker-1.9 version.  I will ping lokesh to see if we can get it back into testing.

Comment 15 Antonio Murdaca 2016-04-28 13:54:53 UTC
Fixed in docker-1.10.3 which is in updates-testing

Comment 16 Fedora Update System 2016-04-28 14:01:10 UTC
docker-1.10.3-15.git964eda6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19

Comment 17 Fedora Update System 2016-05-02 09:22:38 UTC
docker-1.10.3-16.gita41254f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 18 Fedora Update System 2016-05-03 09:27:30 UTC
docker-1.10.3-16.gita41254f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 19 Fedora Update System 2016-05-05 09:41:40 UTC
docker-1.10.3-17.gitbba2d6d.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c7e793ee33

Comment 20 Fedora Update System 2016-05-05 14:05:14 UTC
docker-1.10.3-18.git667d6d1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 21 Fedora Update System 2016-05-06 20:56:45 UTC
docker-1.10.3-18.git667d6d1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 22 Fedora Update System 2016-05-20 10:35:42 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 23 Fedora Update System 2016-05-21 02:26:10 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 24 Fedora Update System 2016-05-24 16:45:26 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 25 Dusty Mabe 2016-05-25 18:25:44 UTC
so are we actually going to release 1.10 to fedora 23 now?

Comment 26 Dusty Mabe 2016-05-25 18:28:50 UTC
Created attachment 1161562 [details]
Seeing this issue with Atomic App on Fedora as well

Comment 27 Daniel Walsh 2016-05-25 21:53:41 UTC
We will if k8s/OpenShift says we can.

Comment 28 Fedora Update System 2016-05-26 05:02:14 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 29 Fedora Update System 2016-05-26 18:28:10 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 30 Dusty Mabe 2016-05-27 21:37:30 UTC
I'm seeing this on RHEL now (on the CDK) - boot up the CDK and run `atomic run projectatomic/mariadb-centos7-atomicapp --mode=fetch`.

[root@rhel-cdk ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@rhel-cdk ~]# rpm -q docker atomic
docker-1.9.1-25.el7.x86_64
atomic-1.9-4.gitff44c6a.el7.x86_64

Comment 31 Dusty Mabe 2016-05-27 22:08:04 UTC
And it was fixed in https://github.com/projectatomic/docker/pull/98 as indicated by Antonio.

Comment 32 Fedora Update System 2016-05-28 03:25:45 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 33 Fedora Update System 2016-05-30 17:36:18 UTC
docker-1.10.3-22.git4158ccc.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 34 Fedora Update System 2016-05-31 09:52:41 UTC
docker-1.10.3-22.git4158ccc.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 35 Fedora Update System 2016-06-01 09:02:00 UTC
docker-1.10.3-23.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d73f74a557

Comment 36 Fedora Update System 2016-06-01 14:53:02 UTC
docker-1.10.3-24.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 37 Fedora Update System 2016-06-02 16:23:05 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 38 Fedora Update System 2016-06-10 02:26:00 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.