Bug 1566455
| Summary: | node can not be started when openstack cloudprovider is enabled. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Johnny Liu <jialiu> | ||||||
| Component: | Installer | Assignee: | Seth Jennings <sjenning> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Johnny Liu <jialiu> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 3.10.0 | CC: | aos-bugs, cshereme, dma, dondavis, hongli, jialiu, jliggitt, jokerman, jwhiting, mmccomas, mtaru, piqin, sgaikwad, tzumainn, vrutkovs, wmeng, xxia | ||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||
| Target Release: | 3.10.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-07-30 19:12:49 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1566451 | ||||||||
| Attachments: |
|
||||||||
|
Description
Johnny Liu
2018-04-12 10:59:54 UTC
This is blocking testing on openstack with cloudprovider enabled. Based on openshift-ansible-3.10.0-0.22.0, change ansible code to workaround Bug 1566814 and Bug 1566406, try again, node still failed to start, but different error. Apr 19 07:00:47 host-172-16-120-48 systemd[1]: Starting OpenShift Node... Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.333833 2423 start_node.go:311] Reading node configuration from /etc/origin/node/bootstrap-node-config.yaml Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.334785 2423 start_node.go:242] Defaulting to the DNSIP config to the node's IP Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.335368 2423 feature_gate.go:190] feature gates: map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true] Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --allow-privileged has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authentication-token-webhook-cache-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-webhook-cache-authorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-webhook-cache-unauthorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13 Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --enable-controller-attach-detach has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --healthz-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-ipc-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-ipc-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-network-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-network-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-pid-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-pid-sources has been deprecated, will be removed in a future version Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --http-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --iptables-masquerade-bit has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --max-pods has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-min-version has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454587 2422 flags.go:27] FLAG: --address="0.0.0.0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454639 2422 flags.go:27] FLAG: --allow-privileged="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454645 2422 flags.go:27] FLAG: --alsologtostderr="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454651 2422 flags.go:27] FLAG: --anonymous-auth="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454655 2422 flags.go:27] FLAG: --application-metrics-count-limit="100" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454660 2422 flags.go:27] FLAG: --authentication-token-webhook="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454663 2422 flags.go:27] FLAG: --authentication-token-webhook-cache-ttl="5m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454669 2422 flags.go:27] FLAG: --authorization-mode="Webhook" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454678 2422 flags.go:27] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454682 2422 flags.go:27] FLAG: --authorization-webhook-cache-unauthorized-ttl="5m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454686 2422 flags.go:27] FLAG: --azure-container-registry-config="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454689 2422 flags.go:27] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454694 2422 flags.go:27] FLAG: --bootstrap-checkpoint-path="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454697 2422 flags.go:27] FLAG: --bootstrap-kubeconfig="/etc/origin/node/bootstrap.kubeconfig" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454702 2422 flags.go:27] FLAG: --cadvisor-port="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454710 2422 flags.go:27] FLAG: --cert-dir="/etc/origin/node/certificates" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454713 2422 flags.go:27] FLAG: --cgroup-driver="systemd" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454717 2422 flags.go:27] FLAG: --cgroup-root="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454720 2422 flags.go:27] FLAG: --cgroups-per-qos="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454724 2422 flags.go:27] FLAG: --chaos-chance="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454730 2422 flags.go:27] FLAG: --client-ca-file="/etc/origin/node/client-ca.crt" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454734 2422 flags.go:27] FLAG: --cloud-config="/etc/origin/cloudprovider/openstack.conf" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454738 2422 flags.go:27] FLAG: --cloud-provider="openstack" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454742 2422 flags.go:27] FLAG: --cluster-dns="[172.16.120.48]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454754 2422 flags.go:27] FLAG: --cluster-domain="cluster.local" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454758 2422 flags.go:27] FLAG: --cni-bin-dir="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454761 2422 flags.go:27] FLAG: --cni-conf-dir="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454765 2422 flags.go:27] FLAG: --config="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454769 2422 flags.go:27] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454773 2422 flags.go:27] FLAG: --container-log-max-files="5" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454777 2422 flags.go:27] FLAG: --container-log-max-size="10Mi" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454780 2422 flags.go:27] FLAG: --container-runtime="docker" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454784 2422 flags.go:27] FLAG: --container-runtime-endpoint="/var/run/dockershim.sock" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454788 2422 flags.go:27] FLAG: --containerd="unix:///var/run/containerd.sock" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454792 2422 flags.go:27] FLAG: --containerized="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454796 2422 flags.go:27] FLAG: --contention-profiling="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454800 2422 flags.go:27] FLAG: --cpu-cfs-quota="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454803 2422 flags.go:27] FLAG: --cpu-manager-policy="none" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454806 2422 flags.go:27] FLAG: --cpu-manager-reconcile-period="10s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454810 2422 flags.go:27] FLAG: --docker="unix:///var/run/docker.sock" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454814 2422 flags.go:27] FLAG: --docker-disable-shared-pid="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454817 2422 flags.go:27] FLAG: --docker-endpoint="unix:///var/run/docker.sock" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454821 2422 flags.go:27] FLAG: --docker-env-metadata-whitelist="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454824 2422 flags.go:27] FLAG: --docker-only="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454828 2422 flags.go:27] FLAG: --docker-root="/var/lib/docker" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454832 2422 flags.go:27] FLAG: --docker-tls="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454835 2422 flags.go:27] FLAG: --docker-tls-ca="ca.pem" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454839 2422 flags.go:27] FLAG: --docker-tls-cert="cert.pem" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454842 2422 flags.go:27] FLAG: --docker-tls-key="key.pem" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454846 2422 flags.go:27] FLAG: --dynamic-config-dir="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454865 2422 flags.go:27] FLAG: --enable-controller-attach-detach="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454869 2422 flags.go:27] FLAG: --enable-custom-metrics="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454872 2422 flags.go:27] FLAG: --enable-debugging-handlers="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454875 2422 flags.go:27] FLAG: --enable-load-reader="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454879 2422 flags.go:27] FLAG: --enable-server="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454882 2422 flags.go:27] FLAG: --enforce-node-allocatable="[pods]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454889 2422 flags.go:27] FLAG: --event-burst="10" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454892 2422 flags.go:27] FLAG: --event-qps="5" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454896 2422 flags.go:27] FLAG: --event-storage-age-limit="default=0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454900 2422 flags.go:27] FLAG: --event-storage-event-limit="default=0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454903 2422 flags.go:27] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454916 2422 flags.go:27] FLAG: --eviction-max-pod-grace-period="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454920 2422 flags.go:27] FLAG: --eviction-minimum-reclaim="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454926 2422 flags.go:27] FLAG: --eviction-pressure-transition-period="5m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454930 2422 flags.go:27] FLAG: --eviction-soft="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454933 2422 flags.go:27] FLAG: --eviction-soft-grace-period="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454937 2422 flags.go:27] FLAG: --exit-on-lock-contention="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454940 2422 flags.go:27] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454944 2422 flags.go:27] FLAG: --experimental-allowed-unsafe-sysctls="[]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454974 2422 flags.go:27] FLAG: --experimental-bootstrap-kubeconfig="/etc/origin/node/bootstrap.kubeconfig" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454979 2422 flags.go:27] FLAG: --experimental-check-node-capabilities-before-mount="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454982 2422 flags.go:27] FLAG: --experimental-dockershim="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454986 2422 flags.go:27] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454989 2422 flags.go:27] FLAG: --experimental-fail-swap-on="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454993 2422 flags.go:27] FLAG: --experimental-kernel-memcg-notification="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454996 2422 flags.go:27] FLAG: --experimental-mounter-path="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455000 2422 flags.go:27] FLAG: --experimental-qos-reserved="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455003 2422 flags.go:27] FLAG: --fail-swap-on="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455007 2422 flags.go:27] FLAG: --feature-gates="RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455017 2422 flags.go:27] FLAG: --file-check-frequency="0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455020 2422 flags.go:27] FLAG: --global-housekeeping-interval="1m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455025 2422 flags.go:27] FLAG: --google-json-key="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455028 2422 flags.go:27] FLAG: --hairpin-mode="promiscuous-bridge" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455032 2422 flags.go:27] FLAG: --healthz-bind-address="127.0.0.1" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455035 2422 flags.go:27] FLAG: --healthz-port="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455039 2422 flags.go:27] FLAG: --help="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455042 2422 flags.go:27] FLAG: --host-ipc-sources="[api,file]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455049 2422 flags.go:27] FLAG: --host-network-sources="[api,file]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455057 2422 flags.go:27] FLAG: --host-pid-sources="[api,file]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455063 2422 flags.go:27] FLAG: --hostname-override="172.16.120.48" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455067 2422 flags.go:27] FLAG: --housekeeping-interval="10s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455070 2422 flags.go:27] FLAG: --http-check-frequency="0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455074 2422 flags.go:27] FLAG: --image-gc-high-threshold="85" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455077 2422 flags.go:27] FLAG: --image-gc-low-threshold="80" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455081 2422 flags.go:27] FLAG: --image-pull-progress-deadline="1m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455085 2422 flags.go:27] FLAG: --image-service-endpoint="/var/run/dockershim.sock" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455088 2422 flags.go:27] FLAG: --iptables-drop-bit="15" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455092 2422 flags.go:27] FLAG: --iptables-masquerade-bit="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455096 2422 flags.go:27] FLAG: --keep-terminated-pod-volumes="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455099 2422 flags.go:27] FLAG: --kube-api-burst="10" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455103 2422 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455107 2422 flags.go:27] FLAG: --kube-api-qps="5" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455110 2422 flags.go:27] FLAG: --kube-reserved="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455114 2422 flags.go:27] FLAG: --kube-reserved-cgroup="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455117 2422 flags.go:27] FLAG: --kubeconfig="/etc/origin/node/node.kubeconfig" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455122 2422 flags.go:27] FLAG: --kubelet-cgroups="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455125 2422 flags.go:27] FLAG: --lock-file="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455128 2422 flags.go:27] FLAG: --log-backtrace-at=":0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455133 2422 flags.go:27] FLAG: --log-cadvisor-usage="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455136 2422 flags.go:27] FLAG: --log-dir="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455140 2422 flags.go:27] FLAG: --log-flush-frequency="5s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455143 2422 flags.go:27] FLAG: --logtostderr="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455147 2422 flags.go:27] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455151 2422 flags.go:27] FLAG: --make-iptables-util-chains="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455155 2422 flags.go:27] FLAG: --manifest-url="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455158 2422 flags.go:27] FLAG: --manifest-url-header="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455165 2422 flags.go:27] FLAG: --master-service-namespace="default" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455169 2422 flags.go:27] FLAG: --max-open-files="1000000" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455175 2422 flags.go:27] FLAG: --max-pods="250" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455179 2422 flags.go:27] FLAG: --maximum-dead-containers="-1" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455183 2422 flags.go:27] FLAG: --maximum-dead-containers-per-container="1" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455187 2422 flags.go:27] FLAG: --minimum-container-ttl-duration="0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455190 2422 flags.go:27] FLAG: --minimum-image-ttl-duration="2m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455194 2422 flags.go:27] FLAG: --network-plugin="cni" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455197 2422 flags.go:27] FLAG: --network-plugin-mtu="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455201 2422 flags.go:27] FLAG: --node-ip="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455204 2422 flags.go:27] FLAG: --node-labels="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455210 2422 flags.go:27] FLAG: --node-status-update-frequency="10s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455214 2422 flags.go:27] FLAG: --non-masquerade-cidr="10.0.0.0/8" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455218 2422 flags.go:27] FLAG: --oom-score-adj="-999" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455222 2422 flags.go:27] FLAG: --pod-cidr="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455225 2422 flags.go:27] FLAG: --pod-infra-container-image="registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455229 2422 flags.go:27] FLAG: --pod-manifest-path="/etc/origin/node/pods" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455233 2422 flags.go:27] FLAG: --pod-max-pids="-1" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455236 2422 flags.go:27] FLAG: --pods-per-core="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455240 2422 flags.go:27] FLAG: --port="10250" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455243 2422 flags.go:27] FLAG: --protect-kernel-defaults="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455247 2422 flags.go:27] FLAG: --provider-id="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455250 2422 flags.go:27] FLAG: --read-only-port="0" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455253 2422 flags.go:27] FLAG: --really-crash-for-testing="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455257 2422 flags.go:27] FLAG: --register-node="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455261 2422 flags.go:27] FLAG: --register-schedulable="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455265 2422 flags.go:27] FLAG: --register-with-taints="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455270 2422 flags.go:27] FLAG: --registry-burst="10" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455274 2422 flags.go:27] FLAG: --registry-qps="5" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455278 2422 flags.go:27] FLAG: --resolv-conf="/etc/resolv.conf" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455281 2422 flags.go:27] FLAG: --rkt-api-endpoint="localhost:15441" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455285 2422 flags.go:27] FLAG: --rkt-path="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455288 2422 flags.go:27] FLAG: --rkt-stage1-image="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455292 2422 flags.go:27] FLAG: --root-dir="/var/lib/origin/openshift.local.volumes" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455296 2422 flags.go:27] FLAG: --rotate-certificates="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455299 2422 flags.go:27] FLAG: --runonce="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455303 2422 flags.go:27] FLAG: --runtime-cgroups="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455306 2422 flags.go:27] FLAG: --runtime-request-timeout="2m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455309 2422 flags.go:27] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455313 2422 flags.go:27] FLAG: --serialize-image-pulls="true" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455316 2422 flags.go:27] FLAG: --stderrthreshold="2" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455320 2422 flags.go:27] FLAG: --storage-driver-buffer-duration="1m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455323 2422 flags.go:27] FLAG: --storage-driver-db="cadvisor" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455328 2422 flags.go:27] FLAG: --storage-driver-host="localhost:8086" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455331 2422 flags.go:27] FLAG: --storage-driver-password="root" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455335 2422 flags.go:27] FLAG: --storage-driver-secure="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455338 2422 flags.go:27] FLAG: --storage-driver-table="stats" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455342 2422 flags.go:27] FLAG: --storage-driver-user="root" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455345 2422 flags.go:27] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455350 2422 flags.go:27] FLAG: --sync-frequency="1m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455353 2422 flags.go:27] FLAG: --system-cgroups="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455356 2422 flags.go:27] FLAG: --system-reserved="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455360 2422 flags.go:27] FLAG: --system-reserved-cgroup="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455363 2422 flags.go:27] FLAG: --tls-cert-file="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455367 2422 flags.go:27] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455401 2422 flags.go:27] FLAG: --tls-min-version="VersionTLS12" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455405 2422 flags.go:27] FLAG: --tls-private-key-file="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455409 2422 flags.go:27] FLAG: --v="5" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455413 2422 flags.go:27] FLAG: --version="false" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455419 2422 flags.go:27] FLAG: --vmodule="" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455423 2422 flags.go:27] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455427 2422 flags.go:27] FLAG: --volume-stats-agg-period="1m0s" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455452 2422 feature_gate.go:226] feature gates: &{{} map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true]} Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.460597 2422 mount_linux.go:211] Detected OS with systemd Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: W0419 07:00:47.460691 2422 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.463353 2422 server.go:376] Version: v1.10.0+b81c8f8 Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.463388 2422 feature_gate.go:226] feature gates: &{{} map[RotateKubeletServerCertificate:true RotateKubeletClientCertificate:true]} Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.742950 2422 server.go:494] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.742993 2422 openstack_instances.go:41] openstack.Instances() called Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.743002 2422 openstack_instances.go:48] Claiming to support Instances Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.746821 2422 metadata.go:152] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.872878 2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1 Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.874582 2422 bootstrap.go:54] Kubeconfig /etc/origin/node/node.kubeconfig exists and is valid, skipping bootstrap Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.875416 2422 certificate_store.go:117] Loading cert/key pair from "/etc/origin/node/certificates/kubelet-client-current.pem". Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.876516 2422 server.go:549] Starting client certificate rotation. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.876529 2422 certificate_manager.go:216] Certificate rotation is enabled. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877009 2422 manager.go:154] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/atomic-openshift-node.service" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877211 2422 certificate_manager.go:360] Certificate expiration is 2020-04-18 07:40:27 +0000 UTC, rotation deadline is 2020-01-09 01:52:01.55566279 +0000 UTC Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877229 2422 certificate_manager.go:221] Waiting 15110h51m13.678435566s for next certificate rotation Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.885839 2422 manager.go:253] Docker not connected: <nil> Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.886356 2422 manager.go:165] Rkt not connected: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.886452 2422 manager.go:176] CRI-O not connected: Get http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.895829 2422 fs.go:142] Filesystem UUIDs: map[b171c7ae-8800-4aeb-835c-14704e1e380c:/dev/vda1 ed49112f-7303-40d3-9e3f-a961f98cf63f:/dev/dm-0] Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.895859 2422 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:19 fsType:tmpfs blockSize:0} /dev/mapper/rhel-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/vda1:{mountpoint:/boot major:252 minor:1 fsType:xfs blockSize:0}] Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.897510 2422 manager.go:227] Machine: {NumCores:2 CpuFrequency:2599996 MemoryCapacity:3974086656 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:de1ef526da6c49fa872b90de9d8a66d1 SystemUUID:5233B8B6-92FD-4FBF-88A8-0A1FB54A4C09 BootID:ce052952-9950-46f9-9ae7-3aeaaebd3cf2 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:19 Capacity:1987043328 Type:vfs Inodes:485118 HasInodes:true} {Device:/dev/mapper/rhel-root DeviceMajor:253 DeviceMinor:0 Capacity:19316867072 Type:vfs Inodes:9437184 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:520785920 Type:vfs Inodes:256000 HasInodes:true}] DiskMap:map[253:2:{Name:dm-2 Major:253 Minor:2 Size:13098811392 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:13098811392 Scheduler:none} 252:0:{Name:vda Major:252 Minor:0 Size:42949672960 Scheduler:mq-deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:19327352832 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:46137344 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:e7:7c:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:4294418432 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899049 2422 manager.go:233] Version: {KernelVersion:3.10.0-862.el7.x86_64 ContainerOsVersion:Red Hat Enterprise Linux Server 7.5 (Maipo) DockerVersion:1.13.1 DockerAPIVersion:1.26 CadvisorVersion: CadvisorRevision:} Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899394 2422 server.go:430] Sending events to api server. Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899441 2422 server.go:613] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899687 2422 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: / Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899697 2422 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/origin/openshift.local.volumes ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true} Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899786 2422 container_manager_linux.go:266] Creating device plugin manager: true Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899794 2422 manager.go:102] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899822 2422 state_mem.go:36] [cpumanager] initializing new in-memory state store Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899866 2422 state_mem.go:87] [cpumanager] updated default cpuset: "" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899876 2422 state_mem.go:95] [cpumanager] updated cpuset assignments: "map[]" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899885 2422 state_file.go:119] [cpumanager] state file: restored state from state file "/var/lib/origin/openshift.local.volumes/cpu_manager_state" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899897 2422 state_file.go:120] [cpumanager] state file: defaultCPUSet: Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899931 2422 oom_linux.go:65] attempting to set "/proc/self/oom_score_adj" to "-999" Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899973 2422 openstack_instances.go:41] openstack.Instances() called Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899981 2422 openstack_instances.go:48] Claiming to support Instances Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899986 2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1 Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899997 2422 server.go:887] Using root directory: /var/lib/origin/openshift.local.volumes Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900006 2422 openstack_instances.go:41] openstack.Instances() called Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900012 2422 openstack_instances.go:48] Claiming to support Instances Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900035 2422 kubelet.go:386] cloud provider determined current node name to be qe-jialiu310-master-etcd-1 Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900041 2422 openstack_instances.go:73] NodeAddresses(qe-jialiu310-master-etcd-1) called Apr 19 07:00:48 host-172-16-120-48 atomic-openshift-node[2422]: F0419 07:00:48.084053 2422 server.go:233] failed to run Kubelet: failed to create kubelet: failed to get the addresses of the current instance from the cloud provider: failed to find object # curl http://169.254.169.254/openstack/2012-08-10/meta_data.json {"uuid": "5233b8b6-92fd-4fbf-88a8-0a1fb54a4c09", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu310-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.48"} This is because the hostname and the name are different: "hostname": "qe-jialiu310-master-etcd-1" "name": "172.16.120.48" 2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1 The cloudprovider code is assuming they are the same as it does a lookup by name using the hostname. https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/openstack/openstack.go#L418-L426 This is a configuration issue in Openstack. The name should not be the IP address. Please configure openstack to get the hostname and name to match in the instance metadata. If you can still recreate this issue even after correcting the Openstack configuration, please reopen. If use the same inventory file to run an installation on the same openstack, 3.9 is going well, but 3.10 is hitting such issue, so I think here is some installer bug. I think to figure out why installer is using "qe-jialiu310-master-etcd-1" as node name, but not the "172.16.120.48" as the node name is the key point to fix this issue. When 3.9 installs successfully, are the resulting Node names IPs or hostnames (i.e. output of "oc get nodes")? (In reply to Seth Jennings from comment #7) > When 3.9 installs successfully, are the resulting Node names IPs or > hostnames (i.e. output of "oc get nodes")? [root@host-172-16-120-137 ~]# openshift version openshift v3.9.27 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.16 [root@host-172-16-120-137 ~]# oc get nodes NAME STATUS ROLES AGE VERSION 172.16.120.137 Ready master 26m v1.9.1+a0ce1bc657 172.16.120.145 Ready compute 26m v1.9.1+a0ce1bc657 [root@host-172-16-120-137 ~]# hostname host-172-16-120-137 [root@host-172-16-120-137 ~]# grep -i nodename /etc/origin/node/node-config.yaml nodeName: 172.16.120.137 [root@host-172-16-120-137 ~]# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json {"uuid": "6a95cdda-18e4-49b7-825a-4b44f56b61e5", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-dma-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.137"} 3.9 node log: May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485003 23320 server.go:359] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf" May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485058 23320 openstack_instances.go:39] openstack.Instances() called May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485068 23320 openstack_instances.go:46] Claiming to support Instances May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.489031 23320 metadata.go:151] Attempting to fetch metadata from http://169.254.169.254/openstack/2012- 08-10/meta_data.json May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.880236 23320 server.go:596] cloud provider determined current node name to be 172.16.120.137 3.10 node log: May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338907 16748 server.go:494] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf" May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338982 16748 openstack_instances.go:41] openstack.Instances() called May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338992 16748 openstack_instances.go:48] Claiming to support Instances May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.348529 16748 metadata.go:152] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.455941 16748 server.go:731] cloud provider determined current node name to be qe-jialiutest-master-etcd-1 According to the above logs, 3.9 cloud provider determined node name to be IP, while 3.10 cloud provider determined node name to be hostname, I am not sure why such difference happened, is it relevant to https://bugzilla.redhat.com/show_bug.cgi?id=1572859#c1 (3.9 nodeName is configurable, but 3.10 does not)? This change was done upstream in this PR: https://github.com/kubernetes/kubernetes/pull/58502 To fix issue: https://github.com/kubernetes/kubernetes/issues/57765 I think the idea here was that the hostname is guaranteed to be a valid nodeName in the kubelet configure where the instance name is not. Additionally, the cloud provider is authoritative wrt the nodeName and --hostname-override is not honored when using cloud provider integration. Additional information: https://github.com/kubernetes/kubernetes/issues/61774#issuecomment-376605627 https://github.com/kubernetes/kubernetes/pull/58114#discussion_r160840393 I think the comes down to, yes, this is a deviation from something that was allowed in the past, but it is not allowed any more. The instance name and hostname need to match. Release note requested, back to CLOSED NOTABUG based on Seth's explanation. https://github.com/openshift/openshift-docs/issues/8651#issuecomment-387725327 *** Bug 1572859 has been marked as a duplicate of this bug. *** Unfortunately even if this bug is closed as NOTABUG, I still can not figure out how to make the installation move on, though I understand Seth's explanation. "qe-jialiu310-master-etcd-1" is my openstack instance hostname, but it is not resolvable, it is only an instance name, it is not possible to get cluster working until figure out some way to make "qe-jialiu310-master-etcd-1" resolved to its instance IP. It isn't a matter of making the hostname resolvable. It is a matter of making the instance name in Openstack and the hostname of the instance the same. (In reply to Seth Jennings from comment #14) > It isn't a matter of making the hostname resolvable. It is a matter of > making the instance name in Openstack and the hostname of the instance the > same. I could make the instance name in Openstack and the hostname of the instance the same, but will hit other issues. Once make the instance name in Openstack and the hostname of the instance the same, node will register the hostname to master, but the hostname is not resolvable, network traffic between master and node would be totally broken, though I do not give a shot yet, if possible, I will try that later, and paste my new "failure" (if no failure, will be a big surprise to me) here. After make the instance name in Openstack and the hostname of the instance the same, installation (openshift-ansible-3.10.0-0.38.0.git.7.848b045.el7.noarch) is failed at the following steps, though node service is started.
TASK [openshift_manage_node : Wait for Node Registration]
<--snip-->
FAILED - RETRYING: Wait for Node Registration (1 retries left).
fatal: [host-8-245-194.host.centralci.eng.rdu2.redhat.com -> host-8-245-194.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 50, "changed": false, "failed": true, "results": {"cmd": "/usr/bin/oc get node host-172-16-120-29 -o json -n default", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): nodes \"host-172-16-120-29\" not found\n", "stdout": ""}, "state": "list"}
# hostname
host-172-16-120-29
# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json
{"uuid": "84fe4239-adc8-49c8-8a13-ca48d96f59ab", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu310-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "qe-jialiu310-master-etcd-1"}
[root@host-172-16-120-29 ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
qe-jialiu310-master-etcd-1 NotReady master 3h v1.10.0+b81c8f8
[root@host-172-16-120-29 ~]# oc get hostsubnet
NAME HOST HOST IP SUBNET EGRESS IPS
qe-jialiu310-master-etcd-1 qe-jialiu310-master-etcd-1 172.16.120.29 10.128.0.0/23 []
node is registering itself as "qe-jialiu310-master-etcd-1", but installer is expecting "host-172-16-120-29" node, so installer exited.
Created attachment 1434254 [details]
installation log with inventory file embedded
Not sure what to say, this is an openstack config issue. Either something with the SDN (neutron or whatever) or cloud-init config in the image being used. The hostname command output, the hostname metadata value, and the host metdata value should all be the same internally resolvable name. When I start an openstack instance, I get this: # hostname node1.lab.variantweb.net # curl -s http://169.254.169.254/openstack/2012-08-10/meta_data.json | python -m json.tool | grep name "hostname": "node1.lab.variantweb.net", "name": "node1.lab.variantweb.net", I understand that maybe there was a why to override this in the past, but some of those mechanisms went away and 3.10 and are not likely to return. Some more color, we document that the nodeName must match the instance name and must be RFC 1123 compliant: https://docs.openshift.com/container-platform/3.9/install_config/configuring_openstack.html#openstack-configuring-nodes-manually While this restriction implies that the instance name should be the hostname, it doesn't explicitly state that. Following comment 18, I did some configuration on my managed openstack to make instance name be resolved to internal IP to meet the prerequisite for openstack cloudprovider - "The hostname command output, the hostname metadata value, and the host metdata value should all be the same internally resolvable name", my cluster finally get working. But just as what Jordan mentioned, if it is not allowed to overwrite node name, I doubt some customer would also hit the same issue like me, maybe we could do some openshift-ansible check in advance to notify user meet these prerequisite. Base the above statement, I removed "testbloker" keyword. (In reply to Johnny Liu from comment #23) > my cluster finally get working. A rpm install on rhel7.5 with openshift-ansible-3.10.0-0.41.0.git.0.88119e4.el7.noarch # openshift version openshift v3.10.0-0.41.0 kubernetes v1.10.0+b81c8f8 etcd 3.2.16 FYI a similar problem appeared during 3.9 -> 3.10 on CentralCI for PAPR tests, we worked it around with setting hostname to internal IP - https://github.com/openshift/openshift-ansible/pull/8273/commits/8be42db576de6f650aed2bebdfaebfc0479a5f2c the breaking 1.10 change upstream has been reverted. the instance name will continue to be used as the kubernetes node name, as in 1.9 picked to openshift in https://github.com/openshift/origin/pull/19730 (In reply to Vadim Rutkovsky from comment #25) > FYI a similar problem appeared during 3.9 -> 3.10 on CentralCI for PAPR > tests, we worked it around with setting hostname to internal IP - > https://github.com/openshift/openshift-ansible/pull/8273/commits/ > 8be42db576de6f650aed2bebdfaebfc0479a5f2c For QE's testing, I have no chance to do this, once setting ansible_host=<internal-IP>, ansible controller host can not connect the remote masters/nodes, and I thought even if I put ansible controller into the same subnet as where master/nodes running, would hit comment #16 Re-test this bug with 3.10.0-0.50.0, PR#19730 mentioned in comment 27 is working well, but will hit the same issue in comment #16. TASK [openshift_manage_node : Wait for Node Registration] <--snip--> FAILED - RETRYING: Wait for Node Registration (1 retries left). fatal: [host-8-250-234.host.centralci.eng.rdu2.redhat.com -> host-8-250-234.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 50, "changed": false, "failed": true, "results": {"cmd": "/usr/bin/oc get node host-172-16-120-55 -o json -n default", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): nodes \"host-172-16-120-55\" not found\n", "stdout": ""}, "state": "list"} # hostname host-172-16-120-55 # curl http://169.254.169.254/openstack/2012-08-10/meta_data.json {"uuid": "0a3d98dd-3241-4b70-8156-6e076ea60aa1", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu3102-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.55"} [root@host-172-16-120-55 ~]# oc get nodes NAME STATUS ROLES AGE VERSION 172.16.120.55 NotReady master 1h v1.10.0+b81c8f8 [root@host-172-16-120-55 ~]# oc get hostsubnet NAME HOST HOST IP SUBNET EGRESS IPS 172.16.120.55 172.16.120.55 172.16.120.55 10.128.0.0/23 [] node is registering itself as "172.16.120.55", but installer is expecting "host-172-16-120-55" node, so installer exited. Today I happened to find an interesting thing. # curl -s http://169.254.169.254/openstack/2012-08-10/meta_data.json | python -m json.tool | grep name "hostname": "qe-jialiustage-master-etcd-nfs-1", "name": "qe_jialiustage_master-etcd-nfs-1", Take note that "_" in "name" is turn into "-" in "hostname". For openstack internal DNS, the "hostname" is resolvable, but not "name". The hostname is guaranteed to conform to the DNS-1123 subdomain spec. That is why https://github.com/kubernetes/kubernetes/pull/58502 in comment #10 is submit into upstream, I think the 58502 PR is more reasonable now. Based on the above analysis and comment #29 (even revert PR#58502, can not resolve all QE's issue), we should re-submit #58502 back (QE already figured out openstack internal DNS configuration to meet the right condition for openstack cloudprovider). Reverting 58502 is still the right thing to do as it has the potential to break users with working deployment and didn't consider version skew among the control plane components. The correct thing to do is to make your instance names RFC1123 compliant as well such that the name == hostname (i.e. no underscores). Because in 3.10 nodeName is not configurable (BZ#1572859), if user want to install a cluster with openstack cloudprovider enable, have to meet the following prerequisite: The hostname command output, the name metadata value (instance name) should all be the same internally resolvable name, and it have to meet RFC1123 compliant (i.e. no underscores). This is working well with 3.10.0-0.58.0 build. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816 |