Bug 1326110

Summary: Unable to create containers with Kubernetes master and Docker 1.9.1-9
Product: [Fedora] Fedora Reporter: Andy Goldstein <agoldste>
Component: dockerAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: adimania, admiller, amurdaca, dustymabe, dwalsh, ichavero, jcajka, jchaloup, lsm5, marianne, miminar, nalin, vbatts
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: docker-1.10.3-24.gitf476348.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-10 02:26:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Seeing this issue with Atomic App on Fedora as well none

Description Andy Goldstein 2016-04-11 20:53:40 UTC
Description of problem: Kubernetes master (commit 391ff9d) is unable to create containers with docker-1.9.1-9


Version-Release number of selected component (if applicable): docker-1.9.1-9.gitee06d03.fc23.x86_64


How reproducible: 100%


Steps to Reproduce:
1. git clone https://github.com/kubernetes/kubernetes to $GOPATH/src/k8s.io/kubernetes
2. cd $GOPATH/src/k8s.io/kubernetes
3. sudo ./hack/local-up-cluster.sh
4. kubectl create -f docs/user-guide/pod.yaml
5. kubectl describe pod/nginx

Actual results: Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: SHM size must be greater then 0"


Expected results: Pod runs


Additional info:

Kubernetes has switched from go-dockerclient to docker's engine-api. As part of this change, the HostConfig's ShmSize is an int64, which means it gets defaulted to 0. go-dockerclient didn't contain this field, so it wasn't getting set. But now that it is defaulted to 0, when combined with https://github.com/projectatomic/docker/blob/fedora-1.9/daemon/daemon_unix.go#L138, you get the error listed above.

It looks like docker 1.10 shouldn't have this problem: https://github.com/projectatomic/docker/blob/fedora-1.10/daemon/daemon_unix.go#L402.

Would it be possible to either get an updated 1.9 or to have 1.10 in Fedora 23?

Comment 1 Antonio Murdaca 2016-04-11 21:07:20 UTC
The change in our projectatomic/docker is a backport from 1.10 - 1.10 changed a bit from 1.9 and that check has been removed. I'm sure it's an error if you provide a negative shmSize, not sure if upstream is a bug or not. Can you actually create this pod if you use 1.10? shmSize should be validated in both 1.9 and 1.10 when is < 0

Comment 2 Andy Goldstein 2016-04-11 21:10:57 UTC
I will test with 1.10 on rawhide later tonight and report back.

However, this pod creates a container that does not have ShmSize set... It's not that Kube is setting it to 0 and we can just fix Kube. We need to figure out how to make sure this will work in Docker instead of it being a validation error.

Comment 3 Antonio Murdaca 2016-04-11 21:13:11 UTC
Found the patch and the login behind the change in 1.10 - will fix this for 1.9. It would be good if you could still test this in 1.10. Thanks!

Comment 4 Andy Goldstein 2016-04-12 00:38:52 UTC
1.10 on rawhide works fine

Comment 5 Antonio Murdaca 2016-04-12 10:01:43 UTC
https://github.com/projectatomic/docker/pull/98

Comment 6 Antonio Murdaca 2016-04-12 17:40:58 UTC
just tested and seems to be working now:

19:40:29 amurdaca at localhost in src/k8s.io/kubernetes ‹master*› cluster/kubectl.sh describe pod/nginx
Name:		nginx
Namespace:	default
Node:		127.0.0.1/127.0.0.1
Start Time:	Tue, 12 Apr 2016 19:37:18 +0200
Labels:		app=nginx
Status:		Running
IP:		172.17.0.2
Controllers:	<none>
Containers:
  nginx:
    Container ID:	docker://dfea384c58a8ca44e72607cb947bfa181abab47bbeffc368e07fcb85c63d6b28
    Image:		nginx
    Image ID:		docker://e32087da8ee6dfa45221c48670fa9475f3d8a53a0e9ccabef4f741c62c77d49b
    Port:		80/TCP
    QoS Tier:
      cpu:			BestEffort
      memory:			BestEffort
    State:			Running
      Started:			Tue, 12 Apr 2016 19:40:22 +0200
    Ready:			True
    Restart Count:		0
    Environment Variables:	<none>
Conditions:
  Type		Status
  Ready 	True 
Volumes:
  default-token-rye32:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-rye32
Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath		Type		Reason		Message
  ---------	--------	-----	----			-------------		--------	------		-------
  3m		3m		1	{default-scheduler }				Normal		Scheduled	Successfully assigned nginx to 127.0.0.1
  3m		3m		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Pulling		pulling image "nginx"
  9s		9s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Pulled		Successfully pulled image "nginx"
  9s		9s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Created		Created container with docker id dfea384c58a8
  8s		8s		1	{kubelet 127.0.0.1}	spec.containers{nginx}	Normal		Started		Started container with docker id dfea384c58a8

Comment 7 Antonio Murdaca 2016-04-12 17:44:29 UTC
tested also the old 1.9 which showed me the error you are reporting. So this fix is working fine now.

Comment 8 Daniel Walsh 2016-04-12 17:48:48 UTC
Antonio which version of docker package is this fixed in?

Comment 9 Antonio Murdaca 2016-04-12 17:50:22 UTC
docker-1.9 in rhel7-1.9 branch -> https://github.com/projectatomic/docker/pull/98
I didn't make a PR against fedora-1.9 branch because we don't seem to be shipping fixes for fedora anymore.

Comment 10 Andy Goldstein 2016-04-12 18:54:53 UTC
We need something in Fedora 23 that isn't broken.

Comment 11 Antonio Murdaca 2016-04-12 19:05:45 UTC
confirmed this is working fine with docker-1.10.3

Comment 12 Daniel Walsh 2016-04-12 19:43:43 UTC
docker-1.10.3 is available for f23, I believe.  Just in updates-testing, because we have been asked not to release it.

Comment 13 Andy Goldstein 2016-04-12 19:48:10 UTC
It's not there yet

Comment 14 Daniel Walsh 2016-04-12 20:03:14 UTC
Well it has been pulled back so that we can update the docker-1.9 version.  I will ping lokesh to see if we can get it back into testing.

Comment 15 Antonio Murdaca 2016-04-28 13:54:53 UTC
Fixed in docker-1.10.3 which is in updates-testing

Comment 16 Fedora Update System 2016-04-28 14:01:10 UTC
docker-1.10.3-15.git964eda6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19

Comment 17 Fedora Update System 2016-05-02 09:22:38 UTC
docker-1.10.3-16.gita41254f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 18 Fedora Update System 2016-05-03 09:27:30 UTC
docker-1.10.3-16.gita41254f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 19 Fedora Update System 2016-05-05 09:41:40 UTC
docker-1.10.3-17.gitbba2d6d.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c7e793ee33

Comment 20 Fedora Update System 2016-05-05 14:05:14 UTC
docker-1.10.3-18.git667d6d1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 21 Fedora Update System 2016-05-06 20:56:45 UTC
docker-1.10.3-18.git667d6d1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 22 Fedora Update System 2016-05-20 10:35:42 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 23 Fedora Update System 2016-05-21 02:26:10 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 24 Fedora Update System 2016-05-24 16:45:26 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 25 Dusty Mabe 2016-05-25 18:25:44 UTC
so are we actually going to release 1.10 to fedora 23 now?

Comment 26 Dusty Mabe 2016-05-25 18:28:50 UTC
Created attachment 1161562 [details]
Seeing this issue with Atomic App on Fedora as well

Comment 27 Daniel Walsh 2016-05-25 21:53:41 UTC
We will if k8s/OpenShift says we can.

Comment 28 Fedora Update System 2016-05-26 05:02:14 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 29 Fedora Update System 2016-05-26 18:28:10 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 30 Dusty Mabe 2016-05-27 21:37:30 UTC
I'm seeing this on RHEL now (on the CDK) - boot up the CDK and run `atomic run projectatomic/mariadb-centos7-atomicapp --mode=fetch`.

[root@rhel-cdk ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@rhel-cdk ~]# rpm -q docker atomic
docker-1.9.1-25.el7.x86_64
atomic-1.9-4.gitff44c6a.el7.x86_64

Comment 31 Dusty Mabe 2016-05-27 22:08:04 UTC
And it was fixed in https://github.com/projectatomic/docker/pull/98 as indicated by Antonio.

Comment 32 Fedora Update System 2016-05-28 03:25:45 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 33 Fedora Update System 2016-05-30 17:36:18 UTC
docker-1.10.3-22.git4158ccc.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 34 Fedora Update System 2016-05-31 09:52:41 UTC
docker-1.10.3-22.git4158ccc.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 35 Fedora Update System 2016-06-01 09:02:00 UTC
docker-1.10.3-23.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d73f74a557

Comment 36 Fedora Update System 2016-06-01 14:53:02 UTC
docker-1.10.3-24.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 37 Fedora Update System 2016-06-02 16:23:05 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 38 Fedora Update System 2016-06-10 02:26:00 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.