Bug 1566455 - node can not be started when openstack cloudprovider is enabled.
Summary: node can not be started when openstack cloudprovider is enabled.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.10.0
Assignee: Seth Jennings
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks: 1566451
TreeView+ depends on / blocked
 
Reported: 2018-04-12 10:59 UTC by Johnny Liu
Modified: 2019-04-14 08:56 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-07-30 19:12:49 UTC
Target Upstream Version:


Attachments (Terms of Use)
installation log with inventory file embedded (1.11 MB, text/plain)
2018-04-12 10:59 UTC, Johnny Liu
no flags Details
installation log with inventory file embedded (1.02 MB, text/plain)
2018-05-10 07:41 UTC, Johnny Liu
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:13:33 UTC
Red Hat Knowledge Base (Solution) 3482061 None None None 2018-06-08 15:50:54 UTC

Description Johnny Liu 2018-04-12 10:59:54 UTC
Created attachment 1420810 [details]
installation log with inventory file embedded

Description of problem:

Version-Release number of the following components:
openshift-ansible-3.10.0-0.16.0.git.0.8925606.el7.noarch
openshift v3.10.0-0.16.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16


How reproducible:
Always

Steps to Reproduce:
1. Trigger an installation on openstack with cloudprovider enabled
2.
3.

Actual results:
job failed at the following task:

TASK [openshift_node : fail] ***************************************************
Thursday 12 April 2018  06:42:05 -0400 (0:00:00.070)       0:13:22.503 ******** 
skipping: [host-8-245-104.host.centralci.eng.rdu2.redhat.com] => {"changed": false, "skip_reason": "Conditional result was False"}
fatal: [host-8-249-83.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"changed": false, "msg": "Node start failed."}

PLAY [Approve any pending CSR requests from inventory nodes] *******************

TASK [Dump all candidate bootstrap hostnames] **********************************
Thursday 12 April 2018  06:42:05 -0400 (0:00:00.072)       0:13:22.575 ******** 
ok: [host-8-245-104.host.centralci.eng.rdu2.redhat.com] => {
    "msg": [
        "host-8-245-104.host.centralci.eng.rdu2.redhat.com", 
        "host-8-249-83.host.centralci.eng.rdu2.redhat.com"
    ]
}

TASK [Find all hostnames for bootstrapping] ************************************
Thursday 12 April 2018  06:42:05 -0400 (0:00:00.046)       0:13:22.622 ******** 
ok: [host-8-245-104.host.centralci.eng.rdu2.redhat.com] => {"ansible_facts": {"l_nodes_to_join": ["host-172-16-120-48", "host-172-16-120-50"]}, "changed": false}

TASK [Dump the bootstrap hostnames] ********************************************
Thursday 12 April 2018  06:42:05 -0400 (0:00:00.079)       0:13:22.701 ******** 
ok: [host-8-245-104.host.centralci.eng.rdu2.redhat.com] => {
    "msg": [
        "host-172-16-120-48", 
        "host-172-16-120-50"
    ]
}

TASK [Approve bootstrap nodes] *************************************************
Thursday 12 April 2018  06:42:05 -0400 (0:00:00.044)       0:13:22.745 ******** 
fatal: [host-8-245-104.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"changed": true, "finished": false, "msg": "Timed out accepting certificate signing requests. Failing as requested.", "nodes": [{"accepted": false, "csrs": {"csr-29bgb": {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-04-12T10:37:04Z", "generateName": "csr-", "name": "csr-29bgb", "namespace": "", "resourceVersion": "1971", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-29bgb", "uid": "70d6da3e-3e3d-11e8-92bf-fa163ee6cd7f"}, "spec": {"groups": ["system:nodes", "system:authenticated"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlVqQ0IrUUlCQURBN01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SWpBZ0JnTlZCQU1UR1hONQpjM1JsYlRwdWIyUmxPakUzTWk0eE5pNHhNakF1TkRnd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DCkFBU3hEREdYdXh3Vi9EUVBWMzdZcEVrUzdEVXZmUDJ1a0xwV2wrQlFLWHNIdHpZVlpxTmx6QUFtbHpyUFZpYjIKZ3lYdzZTMHBNelZvalNyZW1EdEozTFhKb0Z3d1dnWUpLb1pJaHZjTkFRa09NVTB3U3pCSkJnTlZIUkVFUWpCQQpnaEpvYjNOMExURTNNaTB4TmkweE1qQXRORGlDQUljRXJCQjRNSWNRL29BQUFBQUFBQUQ0Rmo3Ly91Yk5mNGNFCnJCRUFBWWNFckJCNE1JY0VDZ2oxYURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTRUZXBZNDJDNXRCU0I5N1QKbWNCUWxBQkszSzFTeWRQN3liUFBCSW1rdERnQ0lER3hXSFhRT3BvRnF3UWY1aGp5dXlWdm5QaHN2bDF6cmF5WApidFI1ZHVRawotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:node:172.16.120.48"}, "status": {"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4VENDQWEyZ0F3SUJBZ0lVSlNaYTQ1U25SY0RrWGszMmZ6ZHo2T1psNWQwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVEl6TlRJNU1UZzFNQjRYRFRFNApNRFF4TWpFd016Y3dNRm9YRFRFNE1EVXhNakV3TXpjd01Gb3dPekVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU0l3SUFZRFZRUURFeGx6ZVhOMFpXMDZibTlrWlRveE56SXVNVFl1TVRJd0xqUTRNRmt3RXdZSEtvWkkKemowQ0FRWUlLb1pJemowREFRY0RRZ0FFc1F3eGw3c2NGZncwRDFkKzJLUkpFdXcxTDN6OXJwQzZWcGZnVUNsNwpCN2MyRldhalpjd0FKcGM2ejFZbTlvTWw4T2t0S1RNMWFJMHEzcGc3U2R5MXlhT0JvRENCblRBT0JnTlZIUThCCkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC9CQUl3QURBZEJnTlYKSFE0RUZnUVU3UDBtRTJLSmtQTy93MU1MTmV4aVJaS09XcTh3U1FZRFZSMFJCRUl3UUlJU2FHOXpkQzB4TnpJdApNVFl0TVRJd0xUUTRnZ0NIQkt3UWVEQ0hFUDZBQUFBQUFBQUErQlkrLy83bXpYK0hCS3dSQUFHSEJLd1FlRENICkJBb0k5V2d3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUcvQUl4dExKVHRDNG9GUlhjakxmL3N2Znh1WHJxdVcKKzBVbWdOb1VsNVBsMndtZVJ0alZuT2NYbVc1Y1hOV0lxV3ZYTWd3ZE10cThQazV0QUswTlUza3MxcTRvcnIweAo0d1JYUEQzeWtsV0F1T2JoY0F2aWVxUU4yM09kMzBDdFdGajNrT2FmWnJFZ1VDOVBOSGlPcHhIRm9XVlA2NjYvClJSd3JIS1VZZTMrcjFsa1lUY3FrejlxcS9CM29rbzM0SDhnYUdaWTJYQS9IT2RwMTJQVFlFSEZWOTQxdkJVdTUKN2hrWENSQnpNNFZMcFVDTGZXdEQvS3BtRjJMYnFza1RKd1JsK0V1MnR1bEdmd0E1b1RsUGhpL2JpUmNzbWVzYgpHMjVUMzhJVkxxQTVzQWdibEoxcVg1RVhqN0oxK0dCUDA0TmdPbTFNUGdMbFpHVUtLVk5keWRVPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==", "conditions": [{"lastUpdateTime": "2018-04-12T10:42:05Z", "message": "This CSR was approved by kubectl certificate approve.", "reason": "KubectlApprove", "type": "Approved"}]}}, "csr-fnwfw": {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-04-12T10:35:53Z", "generateName": "csr-", "name": "csr-fnwfw", "namespace": "", "resourceVersion": "1973", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-fnwfw", "uid": "46fc5783-3e3d-11e8-92bf-fa163ee6cd7f"}, "spec": {"groups": ["system:masters", "system:cluster-admins", "system:authenticated"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlVqQ0IrUUlCQURBN01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SWpBZ0JnTlZCQU1UR1hONQpjM1JsYlRwdWIyUmxPakUzTWk0eE5pNHhNakF1TkRnd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DCkFBU2JTcFovVmdnbU5Ldm1kbjVwYXlZYTkzYXJCU1JmeEIzOElTNFBQaTE0OUliV0NhajNmMnJrYlNENzVvMlgKeEdVdXFrWk50VzBDbHNMVXNxYkJjcWtmb0Z3d1dnWUpLb1pJaHZjTkFRa09NVTB3U3pCSkJnTlZIUkVFUWpCQQpnaEpvYjNOMExURTNNaTB4TmkweE1qQXRORGlDQUljRXJCQjRNSWNRL29BQUFBQUFBQUQ0Rmo3Ly91Yk5mNGNFCnJCRUFBWWNFckJCNE1JY0VDZ2oxYURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBMjE5YllUdEl1S0EyempyTlMKbWpmVWlpRUEwcjJvSGwvdTFvNjR1a056M0FJaEFKenlvYlJmdWhWUU0rT1VZekJsbVBxZlpLa2FLNUcwQWI3aAp2Ync1NllXWAotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:admin"}, "status": {"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4VENDQWEyZ0F3SUJBZ0lVS2wycktMTktJTjRKY3FyQlhRZ3Y4dnU3dFBVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVEl6TlRJNU1UZzFNQjRYRFRFNApNRFF4TWpFd016Y3dNRm9YRFRFNE1EVXhNakV3TXpjd01Gb3dPekVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU0l3SUFZRFZRUURFeGx6ZVhOMFpXMDZibTlrWlRveE56SXVNVFl1TVRJd0xqUTRNRmt3RXdZSEtvWkkKemowQ0FRWUlLb1pJemowREFRY0RRZ0FFbTBxV2YxWUlKalNyNW5aK2FXc21HdmQycXdVa1g4UWQvQ0V1RHo0dAplUFNHMWdtbzkzOXE1RzBnKythTmw4UmxMcXBHVGJWdEFwYkMxTEttd1hLcEg2T0JvRENCblRBT0JnTlZIUThCCkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC9CQUl3QURBZEJnTlYKSFE0RUZnUVVoVVl4bDJDcy8vNUZFekJDWnVZRW5vc2huQkV3U1FZRFZSMFJCRUl3UUlJU2FHOXpkQzB4TnpJdApNVFl0TVRJd0xUUTRnZ0NIQkt3UWVEQ0hFUDZBQUFBQUFBQUErQlkrLy83bXpYK0hCS3dSQUFHSEJLd1FlRENICkJBb0k5V2d3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUNHcmlVRm9HT3I3UUNNUGhaY0w4Tk8vRlZkTkJSL2QKSkYxNklGUHFhNitmYitRb2xtTWNrazFyUTdHUHNUTzJ4ZENHMU16NmhWWlpiV0VDVnBiaTlSS3hLWlNmRXNvagordDZneFY5UkN5dUhPREFwVENwQVNROWZ2cFE1RStHQWhKTmQ4V3BaeXcvUkJOamRVWVcyVllNTkRFSVZBM3kvCjJ5S1l6OUkxeC9HWngybkxRY0NZeWE0MnNVUkFGNEpEa09ndnIzTzhwV2g2dXdXUVc2anZCUDBTZWZ0bkNUT1QKN08zV2Z4aHRYalFiK01RcWlCc0lMV2pSRGJOd1A3alJSdWVsR1VyN20vYUEwYUtpRDRrTWwxdUVpbU93QXFTRwpBa3VIajBzeW5wTFRoMzlpZTRRSmhIaUdDWWFXdjVYYXQzWXpXbyszZFdTZndTZlNKRFhzWk5BPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==", "conditions": [{"lastUpdateTime": "2018-04-12T10:42:05Z", "message": "This CSR was approved by kubectl certificate approve.", "reason": "KubectlApprove", "type": "Approved"}]}}}, "denied": false, "name": "host-172-16-120-48"}, {"accepted": false, "csrs": {}, "denied": false, "name": "host-172-16-120-50"}], "results": [{"cmd": "/usr/bin/oc adm certificate approve csr-29bgb", "results": {}, "returncode": 0}, {"cmd": "/usr/bin/oc adm certificate approve csr-fnwfw", "results": {}, "returncode": 0}], "state": "approve", "timeout": true}
...ignoring

TASK [Get CSRs] ****************************************************************
Thursday 12 April 2018  06:43:19 -0400 (0:01:13.719)       0:14:36.465 ******** 
changed: [host-8-245-104.host.centralci.eng.rdu2.redhat.com] => {"changed": true, "cmd": ["oc", "describe", "csr", "--config=/etc/origin/master/admin.kubeconfig"], "delta": "0:00:00.281166", "end": "2018-04-12 06:43:17.622668", "rc": 0, "start": "2018-04-12 06:43:17.341502", "stderr": "", "stderr_lines": [], "stdout": "Name:               csr-29bgb\nLabels:             <none>\nAnnotations:        <none>\nCreationTimestamp:  Thu, 12 Apr 2018 06:37:04 -0400\nRequesting User:    system:node:172.16.120.48\nStatus:             Approved,Issued\nSubject:\n  Common Name:    system:node:172.16.120.48\n  Serial Number:  \n  Organization:   system:nodes\nSubject Alternative Names:\n         DNS Names:     host-172-16-120-48\n                        \n         IP Addresses:  172.16.120.48\n                        fe80::f816:3eff:fee6:cd7f\n                        172.17.0.1\n                        172.16.120.48\n                        10.8.245.104\nEvents:  <none>\n\n\nName:               csr-fnwfw\nLabels:             <none>\nAnnotations:        <none>\nCreationTimestamp:  Thu, 12 Apr 2018 06:35:53 -0400\nRequesting User:    system:admin\nStatus:             Approved,Issued\nSubject:\n  Common Name:    system:node:172.16.120.48\n  Serial Number:  \n  Organization:   system:nodes\nSubject Alternative Names:\n         DNS Names:     host-172-16-120-48\n                        \n         IP Addresses:  172.16.120.48\n                        fe80::f816:3eff:fee6:cd7f\n                        172.17.0.1\n                        172.16.120.48\n                        10.8.245.104\nEvents:  <none>\n\n\nName:               csr-xlxg6\nLabels:             <none>\nAnnotations:        <none>\nCreationTimestamp:  Thu, 12 Apr 2018 06:35:53 -0400\nRequesting User:    system:admin\nStatus:             Approved,Issued\nSubject:\n         Common Name:    system:node:172.16.120.48\n         Serial Number:  \n         Organization:   system:nodes\nEvents:  <none>\n\n\nName:               node-csr-TbCPVaiYvYL7tljRuJZUTphYzIyob-ZyXzTAo-D74Z0\nLabels:             <none>\nAnnotations:        <none>\nCreationTimestamp:  Thu, 12 Apr 2018 06:37:03 -0400\nRequesting User:    system:serviceaccount:openshift-infra:node-bootstrapper\nStatus:             Pending\nSubject:\n         Common Name:    system:node:172.16.120.50\n         Serial Number:  \n         Organization:   system:nodes\nEvents:  <none>", "stdout_lines": ["Name:               csr-29bgb", "Labels:             <none>", "Annotations:        <none>", "CreationTimestamp:  Thu, 12 Apr 2018 06:37:04 -0400", "Requesting User:    system:node:172.16.120.48", "Status:             Approved,Issued", "Subject:", "  Common Name:    system:node:172.16.120.48", "  Serial Number:  ", "  Organization:   system:nodes", "Subject Alternative Names:", "         DNS Names:     host-172-16-120-48", "                        ", "         IP Addresses:  172.16.120.48", "                        fe80::f816:3eff:fee6:cd7f", "                        172.17.0.1", "                        172.16.120.48", "                        10.8.245.104", "Events:  <none>", "", "", "Name:               csr-fnwfw", "Labels:             <none>", "Annotations:        <none>", "CreationTimestamp:  Thu, 12 Apr 2018 06:35:53 -0400", "Requesting User:    system:admin", "Status:             Approved,Issued", "Subject:", "  Common Name:    system:node:172.16.120.48", "  Serial Number:  ", "  Organization:   system:nodes", "Subject Alternative Names:", "         DNS Names:     host-172-16-120-48", "                        ", "         IP Addresses:  172.16.120.48", "                        fe80::f816:3eff:fee6:cd7f", "                        172.17.0.1", "                        172.16.120.48", "                        10.8.245.104", "Events:  <none>", "", "", "Name:               csr-xlxg6", "Labels:             <none>", "Annotations:        <none>", "CreationTimestamp:  Thu, 12 Apr 2018 06:35:53 -0400", "Requesting User:    system:admin", "Status:             Approved,Issued", "Subject:", "         Common Name:    system:node:172.16.120.48", "         Serial Number:  ", "         Organization:   system:nodes", "Events:  <none>", "", "", "Name:               node-csr-TbCPVaiYvYL7tljRuJZUTphYzIyob-ZyXzTAo-D74Z0", "Labels:             <none>", "Annotations:        <none>", "CreationTimestamp:  Thu, 12 Apr 2018 06:37:03 -0400", "Requesting User:    system:serviceaccount:openshift-infra:node-bootstrapper", "Status:             Pending", "Subject:", "         Common Name:    system:node:172.16.120.50", "         Serial Number:  ", "         Organization:   system:nodes", "Events:  <none>"]}

TASK [Report approval errors] **************************************************
Thursday 12 April 2018  06:43:20 -0400 (0:00:00.463)       0:14:36.928 ******** 
fatal: [host-8-245-104.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"changed": false, "msg": "Node approval failed"}
	to retry, use: --limit @/home/slave1/workspace/Launch Environment Flexy/private-openshift-ansible/playbooks/deploy_cluster.retry


The master node is started successfully, but the compute node failed.
# journalctl -f -u atomic-openshift-node.service 
-- Logs begin at Thu 2018-04-12 06:24:32 EDT. --
Apr 12 06:57:25 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:25.833489   17864 server.go:237] Version: v1.9.1+a0ce1bc657
Apr 12 06:57:25 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:25.833558   17864 feature_gate.go:220] feature gates: &{{} map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true]}
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.038571   17864 server.go:360] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf"
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.038647   17864 openstack_instances.go:39] openstack.Instances() called
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.038657   17864 openstack_instances.go:46] Claiming to support Instances
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.043544   17864 metadata.go:151] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.345379   17864 server.go:601] cloud provider determined current node name to be 172.16.120.50
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.345453   17864 bootstrap.go:58] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.362417   17864 csr.go:105] csr for this node already exists, reusing
Apr 12 06:57:26 host-172-16-120-50 atomic-openshift-node[17864]: I0412 06:57:26.364713   17864 csr.go:113] csr for this node is still valid


Seen from the log, no useful message, and hang there until timeout and restart in a loop.


Expected results:
installation is passed.

Additional info:
If not enabled openstack cloudprovider, no such issue.

Comment 1 Johnny Liu 2018-04-12 11:00:56 UTC
This is blocking testing on openstack with cloudprovider enabled.

Comment 2 Johnny Liu 2018-04-19 11:02:22 UTC
Based on openshift-ansible-3.10.0-0.22.0, change ansible code to workaround Bug 1566814 and Bug 1566406, try again, node still failed to start, but different error.

Apr 19 07:00:47 host-172-16-120-48 systemd[1]: Starting OpenShift Node...
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.333833    2423 start_node.go:311] Reading node configuration from /etc/origin/node/bootstrap-node-config.yaml
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.334785    2423 start_node.go:242] Defaulting to the DNSIP config to the node's IP
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.335368    2423 feature_gate.go:190] feature gates: map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true]
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --allow-privileged has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authentication-token-webhook-cache-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-webhook-cache-authorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --authorization-webhook-cache-unauthorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --enable-controller-attach-detach has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --healthz-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-ipc-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-ipc-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-network-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-network-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-pid-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --host-pid-sources has been deprecated, will be removed in a future version
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --http-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --iptables-masquerade-bit has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --max-pods has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-min-version has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454587    2422 flags.go:27] FLAG: --address="0.0.0.0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454639    2422 flags.go:27] FLAG: --allow-privileged="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454645    2422 flags.go:27] FLAG: --alsologtostderr="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454651    2422 flags.go:27] FLAG: --anonymous-auth="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454655    2422 flags.go:27] FLAG: --application-metrics-count-limit="100"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454660    2422 flags.go:27] FLAG: --authentication-token-webhook="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454663    2422 flags.go:27] FLAG: --authentication-token-webhook-cache-ttl="5m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454669    2422 flags.go:27] FLAG: --authorization-mode="Webhook"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454678    2422 flags.go:27] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454682    2422 flags.go:27] FLAG: --authorization-webhook-cache-unauthorized-ttl="5m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454686    2422 flags.go:27] FLAG: --azure-container-registry-config=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454689    2422 flags.go:27] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454694    2422 flags.go:27] FLAG: --bootstrap-checkpoint-path=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454697    2422 flags.go:27] FLAG: --bootstrap-kubeconfig="/etc/origin/node/bootstrap.kubeconfig"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454702    2422 flags.go:27] FLAG: --cadvisor-port="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454710    2422 flags.go:27] FLAG: --cert-dir="/etc/origin/node/certificates"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454713    2422 flags.go:27] FLAG: --cgroup-driver="systemd"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454717    2422 flags.go:27] FLAG: --cgroup-root=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454720    2422 flags.go:27] FLAG: --cgroups-per-qos="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454724    2422 flags.go:27] FLAG: --chaos-chance="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454730    2422 flags.go:27] FLAG: --client-ca-file="/etc/origin/node/client-ca.crt"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454734    2422 flags.go:27] FLAG: --cloud-config="/etc/origin/cloudprovider/openstack.conf"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454738    2422 flags.go:27] FLAG: --cloud-provider="openstack"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454742    2422 flags.go:27] FLAG: --cluster-dns="[172.16.120.48]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454754    2422 flags.go:27] FLAG: --cluster-domain="cluster.local"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454758    2422 flags.go:27] FLAG: --cni-bin-dir=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454761    2422 flags.go:27] FLAG: --cni-conf-dir=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454765    2422 flags.go:27] FLAG: --config=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454769    2422 flags.go:27] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454773    2422 flags.go:27] FLAG: --container-log-max-files="5"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454777    2422 flags.go:27] FLAG: --container-log-max-size="10Mi"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454780    2422 flags.go:27] FLAG: --container-runtime="docker"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454784    2422 flags.go:27] FLAG: --container-runtime-endpoint="/var/run/dockershim.sock"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454788    2422 flags.go:27] FLAG: --containerd="unix:///var/run/containerd.sock"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454792    2422 flags.go:27] FLAG: --containerized="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454796    2422 flags.go:27] FLAG: --contention-profiling="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454800    2422 flags.go:27] FLAG: --cpu-cfs-quota="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454803    2422 flags.go:27] FLAG: --cpu-manager-policy="none"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454806    2422 flags.go:27] FLAG: --cpu-manager-reconcile-period="10s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454810    2422 flags.go:27] FLAG: --docker="unix:///var/run/docker.sock"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454814    2422 flags.go:27] FLAG: --docker-disable-shared-pid="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454817    2422 flags.go:27] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454821    2422 flags.go:27] FLAG: --docker-env-metadata-whitelist=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454824    2422 flags.go:27] FLAG: --docker-only="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454828    2422 flags.go:27] FLAG: --docker-root="/var/lib/docker"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454832    2422 flags.go:27] FLAG: --docker-tls="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454835    2422 flags.go:27] FLAG: --docker-tls-ca="ca.pem"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454839    2422 flags.go:27] FLAG: --docker-tls-cert="cert.pem"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454842    2422 flags.go:27] FLAG: --docker-tls-key="key.pem"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454846    2422 flags.go:27] FLAG: --dynamic-config-dir=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454865    2422 flags.go:27] FLAG: --enable-controller-attach-detach="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454869    2422 flags.go:27] FLAG: --enable-custom-metrics="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454872    2422 flags.go:27] FLAG: --enable-debugging-handlers="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454875    2422 flags.go:27] FLAG: --enable-load-reader="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454879    2422 flags.go:27] FLAG: --enable-server="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454882    2422 flags.go:27] FLAG: --enforce-node-allocatable="[pods]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454889    2422 flags.go:27] FLAG: --event-burst="10"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454892    2422 flags.go:27] FLAG: --event-qps="5"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454896    2422 flags.go:27] FLAG: --event-storage-age-limit="default=0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454900    2422 flags.go:27] FLAG: --event-storage-event-limit="default=0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454903    2422 flags.go:27] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454916    2422 flags.go:27] FLAG: --eviction-max-pod-grace-period="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454920    2422 flags.go:27] FLAG: --eviction-minimum-reclaim=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454926    2422 flags.go:27] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454930    2422 flags.go:27] FLAG: --eviction-soft=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454933    2422 flags.go:27] FLAG: --eviction-soft-grace-period=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454937    2422 flags.go:27] FLAG: --exit-on-lock-contention="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454940    2422 flags.go:27] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454944    2422 flags.go:27] FLAG: --experimental-allowed-unsafe-sysctls="[]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454974    2422 flags.go:27] FLAG: --experimental-bootstrap-kubeconfig="/etc/origin/node/bootstrap.kubeconfig"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454979    2422 flags.go:27] FLAG: --experimental-check-node-capabilities-before-mount="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454982    2422 flags.go:27] FLAG: --experimental-dockershim="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454986    2422 flags.go:27] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454989    2422 flags.go:27] FLAG: --experimental-fail-swap-on="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454993    2422 flags.go:27] FLAG: --experimental-kernel-memcg-notification="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.454996    2422 flags.go:27] FLAG: --experimental-mounter-path=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455000    2422 flags.go:27] FLAG: --experimental-qos-reserved=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455003    2422 flags.go:27] FLAG: --fail-swap-on="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455007    2422 flags.go:27] FLAG: --feature-gates="RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455017    2422 flags.go:27] FLAG: --file-check-frequency="0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455020    2422 flags.go:27] FLAG: --global-housekeeping-interval="1m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455025    2422 flags.go:27] FLAG: --google-json-key=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455028    2422 flags.go:27] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455032    2422 flags.go:27] FLAG: --healthz-bind-address="127.0.0.1"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455035    2422 flags.go:27] FLAG: --healthz-port="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455039    2422 flags.go:27] FLAG: --help="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455042    2422 flags.go:27] FLAG: --host-ipc-sources="[api,file]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455049    2422 flags.go:27] FLAG: --host-network-sources="[api,file]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455057    2422 flags.go:27] FLAG: --host-pid-sources="[api,file]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455063    2422 flags.go:27] FLAG: --hostname-override="172.16.120.48"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455067    2422 flags.go:27] FLAG: --housekeeping-interval="10s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455070    2422 flags.go:27] FLAG: --http-check-frequency="0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455074    2422 flags.go:27] FLAG: --image-gc-high-threshold="85"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455077    2422 flags.go:27] FLAG: --image-gc-low-threshold="80"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455081    2422 flags.go:27] FLAG: --image-pull-progress-deadline="1m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455085    2422 flags.go:27] FLAG: --image-service-endpoint="/var/run/dockershim.sock"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455088    2422 flags.go:27] FLAG: --iptables-drop-bit="15"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455092    2422 flags.go:27] FLAG: --iptables-masquerade-bit="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455096    2422 flags.go:27] FLAG: --keep-terminated-pod-volumes="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455099    2422 flags.go:27] FLAG: --kube-api-burst="10"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455103    2422 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455107    2422 flags.go:27] FLAG: --kube-api-qps="5"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455110    2422 flags.go:27] FLAG: --kube-reserved=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455114    2422 flags.go:27] FLAG: --kube-reserved-cgroup=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455117    2422 flags.go:27] FLAG: --kubeconfig="/etc/origin/node/node.kubeconfig"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455122    2422 flags.go:27] FLAG: --kubelet-cgroups=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455125    2422 flags.go:27] FLAG: --lock-file=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455128    2422 flags.go:27] FLAG: --log-backtrace-at=":0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455133    2422 flags.go:27] FLAG: --log-cadvisor-usage="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455136    2422 flags.go:27] FLAG: --log-dir=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455140    2422 flags.go:27] FLAG: --log-flush-frequency="5s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455143    2422 flags.go:27] FLAG: --logtostderr="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455147    2422 flags.go:27] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455151    2422 flags.go:27] FLAG: --make-iptables-util-chains="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455155    2422 flags.go:27] FLAG: --manifest-url=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455158    2422 flags.go:27] FLAG: --manifest-url-header=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455165    2422 flags.go:27] FLAG: --master-service-namespace="default"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455169    2422 flags.go:27] FLAG: --max-open-files="1000000"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455175    2422 flags.go:27] FLAG: --max-pods="250"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455179    2422 flags.go:27] FLAG: --maximum-dead-containers="-1"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455183    2422 flags.go:27] FLAG: --maximum-dead-containers-per-container="1"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455187    2422 flags.go:27] FLAG: --minimum-container-ttl-duration="0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455190    2422 flags.go:27] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455194    2422 flags.go:27] FLAG: --network-plugin="cni"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455197    2422 flags.go:27] FLAG: --network-plugin-mtu="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455201    2422 flags.go:27] FLAG: --node-ip=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455204    2422 flags.go:27] FLAG: --node-labels=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455210    2422 flags.go:27] FLAG: --node-status-update-frequency="10s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455214    2422 flags.go:27] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455218    2422 flags.go:27] FLAG: --oom-score-adj="-999"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455222    2422 flags.go:27] FLAG: --pod-cidr=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455225    2422 flags.go:27] FLAG: --pod-infra-container-image="registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455229    2422 flags.go:27] FLAG: --pod-manifest-path="/etc/origin/node/pods"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455233    2422 flags.go:27] FLAG: --pod-max-pids="-1"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455236    2422 flags.go:27] FLAG: --pods-per-core="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455240    2422 flags.go:27] FLAG: --port="10250"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455243    2422 flags.go:27] FLAG: --protect-kernel-defaults="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455247    2422 flags.go:27] FLAG: --provider-id=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455250    2422 flags.go:27] FLAG: --read-only-port="0"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455253    2422 flags.go:27] FLAG: --really-crash-for-testing="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455257    2422 flags.go:27] FLAG: --register-node="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455261    2422 flags.go:27] FLAG: --register-schedulable="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455265    2422 flags.go:27] FLAG: --register-with-taints=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455270    2422 flags.go:27] FLAG: --registry-burst="10"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455274    2422 flags.go:27] FLAG: --registry-qps="5"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455278    2422 flags.go:27] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455281    2422 flags.go:27] FLAG: --rkt-api-endpoint="localhost:15441"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455285    2422 flags.go:27] FLAG: --rkt-path=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455288    2422 flags.go:27] FLAG: --rkt-stage1-image=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455292    2422 flags.go:27] FLAG: --root-dir="/var/lib/origin/openshift.local.volumes"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455296    2422 flags.go:27] FLAG: --rotate-certificates="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455299    2422 flags.go:27] FLAG: --runonce="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455303    2422 flags.go:27] FLAG: --runtime-cgroups=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455306    2422 flags.go:27] FLAG: --runtime-request-timeout="2m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455309    2422 flags.go:27] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455313    2422 flags.go:27] FLAG: --serialize-image-pulls="true"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455316    2422 flags.go:27] FLAG: --stderrthreshold="2"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455320    2422 flags.go:27] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455323    2422 flags.go:27] FLAG: --storage-driver-db="cadvisor"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455328    2422 flags.go:27] FLAG: --storage-driver-host="localhost:8086"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455331    2422 flags.go:27] FLAG: --storage-driver-password="root"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455335    2422 flags.go:27] FLAG: --storage-driver-secure="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455338    2422 flags.go:27] FLAG: --storage-driver-table="stats"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455342    2422 flags.go:27] FLAG: --storage-driver-user="root"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455345    2422 flags.go:27] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455350    2422 flags.go:27] FLAG: --sync-frequency="1m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455353    2422 flags.go:27] FLAG: --system-cgroups=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455356    2422 flags.go:27] FLAG: --system-reserved=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455360    2422 flags.go:27] FLAG: --system-reserved-cgroup=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455363    2422 flags.go:27] FLAG: --tls-cert-file=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455367    2422 flags.go:27] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455401    2422 flags.go:27] FLAG: --tls-min-version="VersionTLS12"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455405    2422 flags.go:27] FLAG: --tls-private-key-file=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455409    2422 flags.go:27] FLAG: --v="5"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455413    2422 flags.go:27] FLAG: --version="false"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455419    2422 flags.go:27] FLAG: --vmodule=""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455423    2422 flags.go:27] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455427    2422 flags.go:27] FLAG: --volume-stats-agg-period="1m0s"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.455452    2422 feature_gate.go:226] feature gates: &{{} map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true]}
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.460597    2422 mount_linux.go:211] Detected OS with systemd
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: W0419 07:00:47.460691    2422 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.463353    2422 server.go:376] Version: v1.10.0+b81c8f8
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.463388    2422 feature_gate.go:226] feature gates: &{{} map[RotateKubeletServerCertificate:true RotateKubeletClientCertificate:true]}
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.742950    2422 server.go:494] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.742993    2422 openstack_instances.go:41] openstack.Instances() called
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.743002    2422 openstack_instances.go:48] Claiming to support Instances
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.746821    2422 metadata.go:152] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.872878    2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.874582    2422 bootstrap.go:54] Kubeconfig /etc/origin/node/node.kubeconfig exists and is valid, skipping bootstrap
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.875416    2422 certificate_store.go:117] Loading cert/key pair from "/etc/origin/node/certificates/kubelet-client-current.pem".
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.876516    2422 server.go:549] Starting client certificate rotation.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.876529    2422 certificate_manager.go:216] Certificate rotation is enabled.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877009    2422 manager.go:154] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/atomic-openshift-node.service"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877211    2422 certificate_manager.go:360] Certificate expiration is 2020-04-18 07:40:27 +0000 UTC, rotation deadline is 2020-01-09 01:52:01.55566279 +0000 UTC
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.877229    2422 certificate_manager.go:221] Waiting 15110h51m13.678435566s for next certificate rotation
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.885839    2422 manager.go:253] Docker not connected: <nil>
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.886356    2422 manager.go:165] Rkt not connected: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.886452    2422 manager.go:176] CRI-O not connected: Get http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.895829    2422 fs.go:142] Filesystem UUIDs: map[b171c7ae-8800-4aeb-835c-14704e1e380c:/dev/vda1 ed49112f-7303-40d3-9e3f-a961f98cf63f:/dev/dm-0]
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.895859    2422 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:19 fsType:tmpfs blockSize:0} /dev/mapper/rhel-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/vda1:{mountpoint:/boot major:252 minor:1 fsType:xfs blockSize:0}]
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.897510    2422 manager.go:227] Machine: {NumCores:2 CpuFrequency:2599996 MemoryCapacity:3974086656 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:de1ef526da6c49fa872b90de9d8a66d1 SystemUUID:5233B8B6-92FD-4FBF-88A8-0A1FB54A4C09 BootID:ce052952-9950-46f9-9ae7-3aeaaebd3cf2 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:19 Capacity:1987043328 Type:vfs Inodes:485118 HasInodes:true} {Device:/dev/mapper/rhel-root DeviceMajor:253 DeviceMinor:0 Capacity:19316867072 Type:vfs Inodes:9437184 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:520785920 Type:vfs Inodes:256000 HasInodes:true}] DiskMap:map[253:2:{Name:dm-2 Major:253 Minor:2 Size:13098811392 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:13098811392 Scheduler:none} 252:0:{Name:vda Major:252 Minor:0 Size:42949672960 Scheduler:mq-deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:19327352832 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:46137344 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:e7:7c:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:4294418432 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899049    2422 manager.go:233] Version: {KernelVersion:3.10.0-862.el7.x86_64 ContainerOsVersion:Red Hat Enterprise Linux Server 7.5 (Maipo) DockerVersion:1.13.1 DockerAPIVersion:1.26 CadvisorVersion: CadvisorRevision:}
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899394    2422 server.go:430] Sending events to api server.
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899441    2422 server.go:613] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899687    2422 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899697    2422 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/origin/openshift.local.volumes ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899786    2422 container_manager_linux.go:266] Creating device plugin manager: true
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899794    2422 manager.go:102] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899822    2422 state_mem.go:36] [cpumanager] initializing new in-memory state store
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899866    2422 state_mem.go:87] [cpumanager] updated default cpuset: ""
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899876    2422 state_mem.go:95] [cpumanager] updated cpuset assignments: "map[]"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899885    2422 state_file.go:119] [cpumanager] state file: restored state from state file "/var/lib/origin/openshift.local.volumes/cpu_manager_state"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899897    2422 state_file.go:120] [cpumanager] state file: defaultCPUSet:
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899931    2422 oom_linux.go:65] attempting to set "/proc/self/oom_score_adj" to "-999"
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899973    2422 openstack_instances.go:41] openstack.Instances() called
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899981    2422 openstack_instances.go:48] Claiming to support Instances
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899986    2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.899997    2422 server.go:887] Using root directory: /var/lib/origin/openshift.local.volumes
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900006    2422 openstack_instances.go:41] openstack.Instances() called
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900012    2422 openstack_instances.go:48] Claiming to support Instances
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900035    2422 kubelet.go:386] cloud provider determined current node name to be qe-jialiu310-master-etcd-1
Apr 19 07:00:47 host-172-16-120-48 atomic-openshift-node[2422]: I0419 07:00:47.900041    2422 openstack_instances.go:73] NodeAddresses(qe-jialiu310-master-etcd-1) called
Apr 19 07:00:48 host-172-16-120-48 atomic-openshift-node[2422]: F0419 07:00:48.084053    2422 server.go:233] failed to run Kubelet: failed to create kubelet: failed to get the addresses of the current instance from the cloud provider: failed to find object


# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json
{"uuid": "5233b8b6-92fd-4fbf-88a8-0a1fb54a4c09", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu310-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.48"}

Comment 4 Seth Jennings 2018-05-06 22:05:12 UTC
This is because the hostname and the name are different:

"hostname": "qe-jialiu310-master-etcd-1"
"name": "172.16.120.48"

2422 server.go:731] cloud provider determined current node name to be qe-jialiu310-master-etcd-1

The cloudprovider code is assuming they are the same as it does a lookup by name using the hostname.

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/openstack/openstack.go#L418-L426

This is a configuration issue in Openstack.  The name should not be the IP address.

Comment 5 Seth Jennings 2018-05-06 22:07:42 UTC
Please configure openstack to get the hostname and name to match in the instance metadata.  If you can still recreate this issue even after correcting the Openstack configuration, please reopen.

Comment 6 Johnny Liu 2018-05-07 03:33:59 UTC
If use the same inventory file to run an installation on the same openstack, 3.9 is going well, but 3.10 is hitting such issue, so I think here is some installer bug. 

I think to figure out why installer is using "qe-jialiu310-master-etcd-1" as node name, but not the "172.16.120.48" as the node name is the key point to fix this issue.

Comment 7 Seth Jennings 2018-05-07 13:51:06 UTC
When 3.9 installs successfully, are the resulting Node names IPs or hostnames (i.e. output of "oc get nodes")?

Comment 8 Johnny Liu 2018-05-08 03:16:56 UTC
(In reply to Seth Jennings from comment #7)
> When 3.9 installs successfully, are the resulting Node names IPs or
> hostnames (i.e. output of "oc get nodes")?

[root@host-172-16-120-137 ~]# openshift version
openshift v3.9.27
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16

[root@host-172-16-120-137 ~]# oc get nodes
NAME             STATUS    ROLES     AGE       VERSION
172.16.120.137   Ready     master    26m       v1.9.1+a0ce1bc657
172.16.120.145   Ready     compute   26m       v1.9.1+a0ce1bc657

[root@host-172-16-120-137 ~]# hostname
host-172-16-120-137

[root@host-172-16-120-137 ~]# grep -i nodename /etc/origin/node/node-config.yaml 
nodeName: 172.16.120.137

[root@host-172-16-120-137 ~]# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json
{"uuid": "6a95cdda-18e4-49b7-825a-4b44f56b61e5", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-dma-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.137"}

Comment 9 Johnny Liu 2018-05-08 06:18:44 UTC
3.9 node log:
May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485003   23320 server.go:359] Successfully initialized cloud provider: "openstack" from the config file:
 "/etc/origin/cloudprovider/openstack.conf"
May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485058   23320 openstack_instances.go:39] openstack.Instances() called
May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.485068   23320 openstack_instances.go:46] Claiming to support Instances
May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.489031   23320 metadata.go:151] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-
08-10/meta_data.json
May 07 21:59:34 host-172-16-120-137 atomic-openshift-node[23320]: I0507 21:59:34.880236   23320 server.go:596] cloud provider determined current node name to be 172.16.120.137


3.10 node log:
May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338907   16748 server.go:494] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf"
May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338982   16748 openstack_instances.go:41] openstack.Instances() called
May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.338992   16748 openstack_instances.go:48] Claiming to support Instances
May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.348529   16748 metadata.go:152] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json
May 08 02:08:37 host-172-16-120-175 atomic-openshift-node[16748]: I0508 02:08:37.455941   16748 server.go:731] cloud provider determined current node name to be qe-jialiutest-master-etcd-1


According to the above logs, 3.9 cloud provider determined node name to be IP, while 3.10 cloud provider determined node name to be hostname, I am not sure why such difference happened, is it relevant to https://bugzilla.redhat.com/show_bug.cgi?id=1572859#c1 (3.9 nodeName is configurable, but 3.10 does not)?

Comment 10 Seth Jennings 2018-05-08 20:09:55 UTC
This change was done upstream in this PR:
https://github.com/kubernetes/kubernetes/pull/58502

To fix issue:
https://github.com/kubernetes/kubernetes/issues/57765

I think the idea here was that the hostname is guaranteed to be a valid nodeName in the kubelet configure where the instance name is not.  Additionally, the cloud provider is authoritative wrt the nodeName and --hostname-override is not honored when using cloud provider integration.

Additional information:
https://github.com/kubernetes/kubernetes/issues/61774#issuecomment-376605627
https://github.com/kubernetes/kubernetes/pull/58114#discussion_r160840393

I think the comes down to, yes, this is a deviation from something that was allowed in the past, but it is not allowed any more.  The instance name and hostname need to match.

Comment 11 Scott Dodson 2018-05-09 12:42:41 UTC
Release note requested, back to CLOSED NOTABUG based on Seth's explanation.

https://github.com/openshift/openshift-docs/issues/8651#issuecomment-387725327

Comment 12 Scott Dodson 2018-05-09 12:43:07 UTC
*** Bug 1572859 has been marked as a duplicate of this bug. ***

Comment 13 Johnny Liu 2018-05-09 15:50:06 UTC
Unfortunately even if this bug is closed as NOTABUG, I still can not figure out how to make the installation move on, though I understand Seth's explanation.


"qe-jialiu310-master-etcd-1" is my openstack instance hostname, but it is not resolvable, it is only an instance name, it is not possible to get cluster working until figure out some way to make "qe-jialiu310-master-etcd-1" resolved to its instance IP.

Comment 14 Seth Jennings 2018-05-09 16:41:26 UTC
It isn't a matter of making the hostname resolvable.  It is a matter of making the instance name in Openstack and the hostname of the instance the same.

Comment 15 Johnny Liu 2018-05-09 16:50:41 UTC
(In reply to Seth Jennings from comment #14)
> It isn't a matter of making the hostname resolvable.  It is a matter of
> making the instance name in Openstack and the hostname of the instance the
> same.

I could make the instance name in Openstack and the hostname of the instance the same, but will hit other issues.

Once make the instance name in Openstack and the hostname of the instance the same, node will register the hostname to master, but the hostname is not resolvable, network traffic between master and node would be totally broken, though I do not give a shot yet, if possible, I will try that later, and paste my new "failure" (if no failure, will be a big surprise to me) here.

Comment 16 Johnny Liu 2018-05-10 07:10:01 UTC
After make the instance name in Openstack and the hostname of the instance the same,  installation (openshift-ansible-3.10.0-0.38.0.git.7.848b045.el7.noarch) is failed at the following steps, though node service is started.

TASK [openshift_manage_node : Wait for Node Registration]
<--snip-->
FAILED - RETRYING: Wait for Node Registration (1 retries left).

fatal: [host-8-245-194.host.centralci.eng.rdu2.redhat.com -> host-8-245-194.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 50, "changed": false, "failed": true, "results": {"cmd": "/usr/bin/oc get node host-172-16-120-29 -o json -n default", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): nodes \"host-172-16-120-29\" not found\n", "stdout": ""}, "state": "list"}


# hostname
host-172-16-120-29

# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json
{"uuid": "84fe4239-adc8-49c8-8a13-ca48d96f59ab", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu310-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "qe-jialiu310-master-etcd-1"}

[root@host-172-16-120-29 ~]# oc get nodes
NAME                         STATUS     ROLES     AGE       VERSION
qe-jialiu310-master-etcd-1   NotReady   master    3h        v1.10.0+b81c8f8

[root@host-172-16-120-29 ~]# oc get hostsubnet
NAME                         HOST                         HOST IP         SUBNET          EGRESS IPS
qe-jialiu310-master-etcd-1   qe-jialiu310-master-etcd-1   172.16.120.29   10.128.0.0/23   []

node is registering itself as "qe-jialiu310-master-etcd-1", but installer is expecting "host-172-16-120-29" node, so installer exited.

Comment 17 Johnny Liu 2018-05-10 07:41:14 UTC
Created attachment 1434254 [details]
installation log with inventory file embedded

Comment 18 Seth Jennings 2018-05-10 18:57:25 UTC
Not sure what to say, this is an openstack config issue.  Either something with the SDN (neutron or whatever) or cloud-init config in the image being used.

The hostname command output, the hostname metadata value, and the host metdata value should all be the same internally resolvable name.

When I start an openstack instance, I get this:

# hostname
node1.lab.variantweb.net

# curl -s http://169.254.169.254/openstack/2012-08-10/meta_data.json | python -m json.tool | grep name
    "hostname": "node1.lab.variantweb.net",
    "name": "node1.lab.variantweb.net",

I understand that maybe there was a why to override this in the past, but some of those mechanisms went away and 3.10 and are not likely to return.

Comment 21 Seth Jennings 2018-05-16 02:22:12 UTC
Some more color, we document that the nodeName must match the instance name and must be RFC 1123 compliant:

https://docs.openshift.com/container-platform/3.9/install_config/configuring_openstack.html#openstack-configuring-nodes-manually

While this restriction implies that the instance name should be the hostname, it doesn't explicitly state that.

Comment 23 Johnny Liu 2018-05-16 03:54:26 UTC
Following comment 18, I did some configuration on my managed openstack to make instance name be resolved to internal IP to meet the prerequisite for openstack cloudprovider - "The hostname command output, the hostname metadata value, and the host metdata value should all be the same internally resolvable name", my cluster finally get working.


But just as what Jordan mentioned, if it is not allowed to overwrite node name, I doubt some customer would also hit the same issue like me, maybe we could do some openshift-ansible check in advance to notify user meet these prerequisite.

Base the above statement, I removed "testbloker" keyword.

Comment 24 Johnny Liu 2018-05-16 08:11:00 UTC
(In reply to Johnny Liu from comment #23)
> my cluster finally get working.

A rpm install on rhel7.5 with openshift-ansible-3.10.0-0.41.0.git.0.88119e4.el7.noarch
# openshift version
openshift v3.10.0-0.41.0
kubernetes v1.10.0+b81c8f8
etcd 3.2.16

Comment 25 Vadim Rutkovsky 2018-05-16 08:56:46 UTC
FYI a similar problem appeared during 3.9 -> 3.10 on CentralCI for PAPR tests, we worked it around with setting hostname to internal IP - https://github.com/openshift/openshift-ansible/pull/8273/commits/8be42db576de6f650aed2bebdfaebfc0479a5f2c

Comment 26 Jordan Liggitt 2018-05-16 17:46:33 UTC
the breaking 1.10 change upstream has been reverted. the instance name will continue to be used as the kubernetes node name, as in 1.9

picked to openshift in https://github.com/openshift/origin/pull/19730

Comment 27 Jordan Liggitt 2018-05-16 21:17:24 UTC
https://github.com/openshift/origin/pull/19730 is merged

Comment 28 Johnny Liu 2018-05-17 06:46:22 UTC
(In reply to Vadim Rutkovsky from comment #25)
> FYI a similar problem appeared during 3.9 -> 3.10 on CentralCI for PAPR
> tests, we worked it around with setting hostname to internal IP -
> https://github.com/openshift/openshift-ansible/pull/8273/commits/
> 8be42db576de6f650aed2bebdfaebfc0479a5f2c

For QE's testing, I have no chance to do this, once setting ansible_host=<internal-IP>, ansible controller host can not connect the remote masters/nodes, and I thought even if I put ansible controller into the same subnet as where master/nodes running, would hit comment #16

Comment 29 Johnny Liu 2018-05-22 08:23:57 UTC
Re-test this bug with 3.10.0-0.50.0, PR#19730 mentioned in comment 27 is working well, but will hit the same issue in comment #16.


TASK [openshift_manage_node : Wait for Node Registration]
<--snip-->
FAILED - RETRYING: Wait for Node Registration (1 retries left).

fatal: [host-8-250-234.host.centralci.eng.rdu2.redhat.com -> host-8-250-234.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 50, "changed": false, "failed": true, "results": {"cmd": "/usr/bin/oc get node host-172-16-120-55 -o json -n default", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): nodes \"host-172-16-120-55\" not found\n", "stdout": ""}, "state": "list"}


# hostname
host-172-16-120-55

# curl http://169.254.169.254/openstack/2012-08-10/meta_data.json
{"uuid": "0a3d98dd-3241-4b70-8156-6e076ea60aa1", "availability_zone": "nova", "keys": [{"data": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV", "type": "ssh", "name": "libra"}], "hostname": "qe-jialiu3102-master-etcd-1", "launch_index": 0, "public_keys": {"libra": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUq7W38xCZ9WGSWCvustaMGMT04tRohw6AKGzI7P7xql5lhCAReyt72n9qWQRZsE1YiCSQuTfXI1oc8NpSM7+lMLwj12G8z3I1YT31JHr9LLYg/XIcExkzfBI920CaS82VqmKOpI9+ARHSJBdIbKRI0f5Y+u4xbc5UzKCJX8jcKGG7nEiw8zm+cvAlfOgssMK+qJppIbVcb2iZNTsw5i2aX6FDMyC+b17DQHzBGpNbhZYxuoERZVRcnYctgIzuo6fD60gniX0fVvrchlOnubB1sRYbloP2r6UE22w/dpLKOFE5i7CA0ZzNBERZ94cIKumIH9MiJs1a6bMe89VOjjNV"}, "name": "172.16.120.55"}

[root@host-172-16-120-55 ~]# oc get nodes
NAME            STATUS     ROLES     AGE       VERSION
172.16.120.55   NotReady   master    1h        v1.10.0+b81c8f8

[root@host-172-16-120-55 ~]# oc get hostsubnet
NAME            HOST            HOST IP         SUBNET          EGRESS IPS
172.16.120.55   172.16.120.55   172.16.120.55   10.128.0.0/23   []

node is registering itself as "172.16.120.55", but installer is expecting "host-172-16-120-55" node, so installer exited.

Comment 30 Johnny Liu 2018-05-24 08:34:09 UTC
Today I happened to find an interesting thing.

# curl -s http://169.254.169.254/openstack/2012-08-10/meta_data.json | python -m json.tool | grep name
    "hostname": "qe-jialiustage-master-etcd-nfs-1",
    "name": "qe_jialiustage_master-etcd-nfs-1",

Take note that "_" in "name" is turn into "-" in "hostname".

For openstack internal DNS, the "hostname" is resolvable, but not "name".  The hostname is guaranteed to conform to the DNS-1123 subdomain spec. That is why https://github.com/kubernetes/kubernetes/pull/58502 in comment #10 is submit into upstream, I think the 58502 PR is more reasonable now.

Based on the above analysis and comment #29 (even revert PR#58502, can not resolve all QE's issue), we should re-submit #58502 back (QE already figured out openstack internal DNS configuration to meet the right condition for openstack cloudprovider).

Comment 31 Seth Jennings 2018-05-24 14:24:45 UTC
Reverting 58502 is still the right thing to do as it has the potential to break users with working deployment and didn't consider version skew among the control plane components.

The correct thing to do is to make your instance names RFC1123 compliant as well such that the name == hostname (i.e. no underscores).

Comment 33 Johnny Liu 2018-06-05 07:41:02 UTC
Because in 3.10 nodeName is not configurable (BZ#1572859), if user want to install a cluster with openstack cloudprovider enable, have to meet the following prerequisite:

The hostname command output, the name metadata value (instance name) should all be the same internally resolvable name, and it have to meet RFC1123 compliant (i.e. no underscores).

This is working well with 3.10.0-0.58.0 build.

Comment 40 errata-xmlrpc 2018-07-30 19:12:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.