Bug 1260587 - Kubelet broke unexpectedly with option '--cloud-provider=openstack'
Kubelet broke unexpectedly with option '--cloud-provider=openstack'
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
3.x
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Jan Safranek
Jianwei Hou
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-07 06:22 EDT by Jianwei Hou
Modified: 2015-11-23 16:16 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 16:16:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jianwei Hou 2015-09-07 06:22:41 EDT
Description of problem:
Start kubelet with '--cloud-provider' and '--cloud-config' options, kubelete broke unexpectedly when the auth-url in cloud config contains 'https'

Version-Release number of selected component (if applicable):
Client Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.371+8cc75a47346b8a", GitCommit:"8cc75a47346b8a93ddfe3127900c02585068019c", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.371+8cc75a47346b8a", GitCommit:"8cc75a47346b8a93ddfe3127900c02585068019c", GitTreeState:"clean"}

How reproducible:
Always

Steps to Reproduce:
1. On openstack(beijing openstack hosted by qe), start one rhel7.1 instance, and run kubernetes with hack/local-up-cluster.sh
2. Create a cloud.conf, follow the instruction of https://github.com/spothanis/kubernetes/blob/c841a20361c86ea94acf5e5aff9f319aaa629e89/examples/mysql-cinder-pd/README.md#prerequisites to set correct auth-url, username, password, region, tenant-id
3. Kill the kubelet process, start it with option '--cloud-provider' and '--cloud-config' given

Actual results:
After step 3:
panic: runtime error: invalid memory address or nil pointer dereference

```
# /root/kubernetes/_output/local/bin/linux/amd64/kubelet --v=3 --chaos-chance=0.0 --container-runtime=docker --hostname-override=127.0.0.1 --address=127.0.0.1 --api-servers=127.0.0.1:8080 --port=10250 --cloud-config=/root/cloud.conf --cloud-provider=openstack

W0907 18:02:29.307956   11829 server.go:512] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0907 18:02:29.308038   11829 server.go:474] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0907 18:02:29.312716   11829 server.go:395] Successfully initialized cloud provider: "openstack" from the config file: "/root/cloud.conf"
I0907 18:02:29.312900   11829 manager.go:127] cAdvisor running in container: "/"
I0907 18:02:29.313502   11829 fs.go:93] Filesystem partitions: map[/dev/mapper/rhel_dhcp--128--79-root:{mountpoint:/ major:253 minor:1} /dev/vda1:{mountpoint:/boot major:252 minor:1}]
I0907 18:02:29.319425   11829 manager.go:158] Machine: {NumCores:1 CpuFrequency:2000000 MemoryCapacity:1929773056 MachineID:3ac9125064c34b8ba24d13c197e4fa9b SystemUUID:3EA49B4D-54B2-4D82-99BD-5FB4407BE9C7 BootID:4c643877-addd-4c5e-919f-6711a362dc97 Filesystems:[{Device:/dev/mapper/rhel_dhcp--128--79-root Capacity:18746441728} {Device:/dev/vda1 Capacity:520794112}] DiskMap:map[253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 252:0:{Name:vda Major:252 Minor:0 Size:21474836480 Scheduler:none} 253:0:{Name:dm-0 Major:253 Minor:0 Size:2147483648 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:18756927488 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:22:c2:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:2147074048 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown}
I0907 18:02:29.357360   11829 manager.go:164] Version: {KernelVersion:3.10.0-229.4.2.el7.x86_64 ContainerOsVersion:Employee SKU DockerVersion:1.7.1 CadvisorVersion:0.16.0}
I0907 18:02:29.357976   11829 server.go:412] Using root directory: /var/lib/kubelet
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x7f1352]

goroutine 1 [running]:
k8s.io/kubernetes/pkg/cloudprovider/providers/openstack.(*OpenStack).Instances(0x0, 0x0, 0x0, 0x9)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/openstack/openstack.go:166 +0x7e2
k8s.io/kubernetes/cmd/kubelet/app.RunKubelet(0xc2082e0300, 0x0, 0x0, 0x0)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:628 +0x12a
k8s.io/kubernetes/cmd/kubelet/app.(*KubeletServer).Run(0xc208200000, 0xc2082e0300, 0x0, 0x0)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:420 +0x82f
main.main()
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:46 +0x95

goroutine 7 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x1d0eea0)
	/root/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/golang/glog.init·1
	/root/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1

goroutine 11 [syscall]:
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
	/usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35

goroutine 18 [sleep]:
k8s.io/kubernetes/pkg/util.Until(0x15efa70, 0x12a05f200, 0xc20802e600)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:117 +0x98
created by k8s.io/kubernetes/pkg/util.InitLogs
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/logs.go:49 +0xab

goroutine 28 [runnable]:
k8s.io/kubernetes/pkg/kubelet/cadvisor.func·001()
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:97
created by k8s.io/kubernetes/pkg/kubelet/cadvisor.(*cadvisorClient).exportHTTP
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:106 +0x4d7
```

Expected results:
cloud provider and cloud config both have to be provided for kubelet to properly cinder volume plugin

Additional info:
Comment 1 Jan Safranek 2015-09-07 09:46:28 EDT
Can you please share your cloud.conf (as private attachment and without password)? Or even give me access to the test machine(s)?
Comment 4 Jan Safranek 2015-09-08 04:56:34 EDT
This PR fixes the crash: https://github.com/kubernetes/kubernetes/pull/13660

Still, kube-apiserver won't start. At least, it will show useful error in its logs:

F0908 08:52:13.313709   12714 server.go:308] Cloud provider could not be initialized: could not init cloud provider "openstack": Post https://<censored>:5000/v2.0/tokens: tls: oversized record received with length 24932
Comment 5 Jianwei Hou 2015-09-10 06:14:40 EDT
Hi, jsafrane, does the username in cloud config has to be an admin?

Running with my configrations, I continuously get
```
I0910 18:05:42.816949    6806 openstack.go:201] Found 4 compute flavors
I0910 18:05:42.817003    6806 openstack.go:202] Claiming to support Instances
E0910 18:05:42.864761    6806 kubelet.go:846] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
```
Comment 6 Jianwei Hou 2015-09-22 02:07:25 EDT
Verified on
Client Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.1113+de597c8dc4da30-dirty", GitCommit:"de597c8dc4da305584cec3347af6bf3e053de59f", GitTreeState:"dirty"}
Server Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.1113+de597c8dc4da30-dirty", GitCommit:"de597c8dc4da305584cec3347af6bf3e053de59f", GitTreeState:"dirty"}

Now only http works, when https is used, error message is more clear now.

Note You need to log in before you can comment on or make changes to this bug.