Attachment 1483324 Details for Bug 1625911 – ansible execution -vvv

ansible execution -vvv

log-ansible (text/plain), 52.94 KB, created by Serena Cortopassi on 2018-09-14 12:07:06 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Serena Cortopassi

Created: 2018-09-14 12:07:06 UTC

Size: 52.94 KB

patch

obsolete

>TASK [Dump all candidate bootstrap hostnames] *********************************************************************************************************************************************
>task path: /home/awx/openshift-ansible-openshift-ansible-3.10.47-1/playbooks/openshift-node/private/join.yml:28
>ok: [red-ose-master01.internal.ose.extrasys.it] => {
>    "msg": [
>        "red-ose-master01.internal.ose.extrasys.it", 
>        "red-ose-node01.internal.ose.extrasys.it", 
>        "red-ose-infra01.internal.ose.extrasys.it"
>    ]
>}
>
>TASK [Find all hostnames for bootstrapping] ***********************************************************************************************************************************************
>task path: /home/awx/openshift-ansible-openshift-ansible-3.10.47-1/playbooks/openshift-node/private/join.yml:32
>ok: [red-ose-master01.internal.ose.extrasys.it] => {
>    "ansible_facts": {
>        "l_nodes_to_join": [
>            "red-ose-master01", 
>            "red-ose-node01", 
>            "red-ose-infra01"
>        ]
>    }, 
>    "changed": false
>}
>
>TASK [Dump the bootstrap hostnames] *******************************************************************************************************************************************************
>task path: /home/awx/openshift-ansible-openshift-ansible-3.10.47-1/playbooks/openshift-node/private/join.yml:36
>ok: [red-ose-master01.internal.ose.extrasys.it] => {
>    "msg": [
>        "red-ose-master01", 
>        "red-ose-node01", 
>        "red-ose-infra01"
>    ]
>}
>
>TASK [Approve node certificates when bootstrapping] ***************************************************************************************************************************************
>
>
>FAILED - RETRYING: Approve node certificates when bootstrapping (1 retries left).Result was: {
>    "all_subjects_found": [
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-infra01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-node01\n"
>    ], 
>    "attempts": 30, 
>    "changed": false, 
>    "client_approve_results": [], 
>    "client_csrs": {
>        "node-csr-fHcxz4629_6-JSLhNYx413PrwaRFQPieNwRWR9iJoMw": "red-ose-infra01", 
>        "node-csr-o3xS6RXO8DREw1AL2Oeb1ZcDAKZir0KGN6YLd2RlwX8": "red-ose-node01"
>    }, 
>    "invocation": {
>        "module_args": {
>            "node_list": [
>                "red-ose-master01", 
>                "red-ose-node01", 
>                "red-ose-infra01"
>            ], 
>            "oc_bin": "oc", 
>            "oc_conf": "/etc/origin/master/admin.kubeconfig"
>        }
>    }, 
>    "msg": "Could not find csr for nodes: red-ose-master01", 
>    "oc_get_nodes": {
>        "apiVersion": "v1", 
>        "items": [
>            {
>                "apiVersion": "v1", 
>                "kind": "Node", 
>                "metadata": {
>                    "annotations": {
>                        "volumes.kubernetes.io/controller-managed-attach-detach": "true"
>                    }, 
>                    "creationTimestamp": "2018-09-14T11:27:47Z", 
>                    "labels": {
>                        "beta.kubernetes.io/arch": "amd64", 
>                        "beta.kubernetes.io/instance-type": "82369357-d10d-4512-b477-d0f6839bd024", 
>                        "beta.kubernetes.io/os": "linux", 
>                        "failure-domain.beta.kubernetes.io/zone": "nova", 
>                        "kubernetes.io/hostname": "red-ose-master01", 
>                        "node-role.kubernetes.io/master": "true"
>                    }, 
>                    "name": "red-ose-master01", 
>                    "namespace": "", 
>                    "resourceVersion": "2170", 
>                    "selfLink": "/api/v1/nodes/red-ose-master01", 
>                    "uid": "34e7397a-b811-11e8-bb4a-fa163e0c2315"
>                }, 
>                "spec": {
>                    "externalID": "e7777176-96a0-46d0-a6e5-2d409f14ae55", 
>                    "providerID": "openstack:///e7777176-96a0-46d0-a6e5-2d409f14ae55"
>                }, 
>                "status": {
>                    "addresses": [
>                        {
>                            "address": "192.168.8.20", 
>                            "type": "InternalIP"
>                        }, 
>                        {
>                            "address": "156.54.176.240", 
>                            "type": "ExternalIP"
>                        }, 
>                        {
>                            "address": "red-ose-master01", 
>                            "type": "Hostname"
>                        }
>                    ], 
>                    "allocatable": {
>                        "cpu": "8", 
>                        "hugepages-1Gi": "0", 
>                        "hugepages-2Mi": "0", 
>                        "memory": "16163948Ki", 
>                        "pods": "250"
>                    }, 
>                    "capacity": {
>                        "cpu": "8", 
>                        "hugepages-1Gi": "0", 
>                        "hugepages-2Mi": "0", 
>                        "memory": "16266348Ki", 
>                        "pods": "250"
>                    }, 
>                    "conditions": [
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient disk space available", 
>                            "reason": "KubeletHasSufficientDisk", 
>                            "status": "False", 
>                            "type": "OutOfDisk"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient memory available", 
>                            "reason": "KubeletHasSufficientMemory", 
>                            "status": "False", 
>                            "type": "MemoryPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has no disk pressure", 
>                            "reason": "KubeletHasNoDiskPressure", 
>                            "status": "False", 
>                            "type": "DiskPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient PID available", 
>                            "reason": "KubeletHasSufficientPID", 
>                            "status": "False", 
>                            "type": "PIDPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", 
>                            "reason": "KubeletNotReady", 
>                            "status": "False", 
>                            "type": "Ready"
>                        }
>                    ], 
>                    "daemonEndpoints": {
>                        "kubeletEndpoint": {
>                            "Port": 10250
>                        }
>                    }, 
>                    "images": [
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-node@sha256:b105ec6800823dc4b1deef0b0fe0abb90afc8e530191606baba014e3d50f1daf", 
>                                "registry.access.redhat.com/openshift3/ose-node:v3.10"
>                            ], 
>                            "sizeBytes": 1268901980
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-control-plane@sha256:3d0b24963b4099bb06e6bf70cd0096c3c332dd814dd6497b60f4ca5902473ca5", 
>                                "registry.access.redhat.com/openshift3/ose-control-plane:v3.10"
>                            ], 
>                            "sizeBytes": 788612067
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/rhel7/etcd@sha256:2129d5aa0655c0eb614a94ee5792dca9721e64217509713a6bcbfd2160f9cbfa", 
>                                "registry.access.redhat.com/rhel7/etcd:3.2.22"
>                            ], 
>                            "sizeBytes": 256010737
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-pod@sha256:6c716eba6a032b5c75690407ef3be8e598047b3b37f3745b71eb67c1a64ee6e0", 
>                                "registry.access.redhat.com/openshift3/ose-pod:v3.10.34"
>                            ], 
>                            "sizeBytes": 214236553
>                        }
>                    ], 
>                    "nodeInfo": {
>                        "architecture": "amd64", 
>                        "bootID": "d70c83c3-9154-4c5d-baf3-ed54b2275f28", 
>                        "containerRuntimeVersion": "docker://1.13.1", 
>                        "kernelVersion": "3.10.0-862.11.6.el7.x86_64", 
>                        "kubeProxyVersion": "v1.10.0+b81c8f8", 
>                        "kubeletVersion": "v1.10.0+b81c8f8", 
>                        "machineID": "3394c16415a04c30b9c81b1aefb5f404", 
>                        "operatingSystem": "linux", 
>                        "osImage": "OpenShift Enterprise", 
>                        "systemUUID": "E7777176-96A0-46D0-A6E5-2D409F14AE55"
>                    }
>                }
>            }
>        ], 
>        "kind": "List", 
>        "metadata": {
>            "resourceVersion": "", 
>            "selfLink": ""
>        }
>    }, 
>    "rc": 0, 
>    "retries": 31, 
>    "server_approve_results": [], 
>    "server_csrs": null, 
>    "state": "unknown", 
>    "unwanted_csrs": [
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:30:17Z", 
>                "generateName": "csr-", 
>                "name": "csr-mmzmj", 
>                "namespace": "", 
>                "resourceVersion": "1102", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-mmzmj", 
>                "uid": "8e469d12-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:nodes", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVE0dkdnNHZqVFZZdEVGUUJoV1k2cjNHU3NYNGtVTDVYQVN4S1ZlVjJ1RmV6MDV6Q3NVYmtDeTdibmQKOUovYktWSmc1cEhlNitTWW55M24rTlk2eEh4RW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFJbjRBK2JnNXZYQVNHMXcKSjk0UGdwOS9xUU56dnJpWkYxc2tvQ0RYWDNPNUFpRUF5cTE3QXBmekpvRVRLYUx5Sm5pZkFDUFUwajFVSmo3aApCekQ2dGk5cVYzbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:node:red-ose-master01"
>            }, 
>            "status": {}
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:27:50Z", 
>                "generateName": "csr-", 
>                "name": "csr-nkgpf", 
>                "namespace": "", 
>                "resourceVersion": "474", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-nkgpf", 
>                "uid": "3651cf91-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:masters", 
>                    "system:cluster-admins", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlINk1JR2dBZ0VBTUQ0eEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVsTUNNR0ExVUVBeE1jYzNsegpkR1Z0T201dlpHVTZjbVZrTFc5elpTMXRZWE4wWlhJd01UQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VICkEwSUFCTjRSWFgrMTI5UXNTMTY1YUsvRVdzRjh6bzNXSTF3MHlmemZmNExzeHF6eENjZi81eldEUlllNGFUVG8KQUtJcGNlTk9VSGprVFk4L2dJSXZVM0lxamVlZ0FEQUtCZ2dxaGtqT1BRUURBZ05KQURCR0FpRUF5RjlsN05DLwpUN2pnMmorQVlLUzVKN2l0ODBrREw5eVo5ZWNxU256SHZuUUNJUURjVVp0dzNrTEJLY2llb3JUcmZBQlFzRUpyCldFdkp4WjU0V3BGYU1idjdOdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "client auth"
>                ], 
>                "username": "system:admin"
>            }, 
>            "status": {
>                "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNlekNDQVdPZ0F3SUJBZ0lVVVZ2TXgvOUlTb1hpWEhmcTdCU3R2MVZDcVI0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVE0yT1RJME16VXlNQjRYRFRFNApNRGt4TkRFeE1qTXdNRm9YRFRFNU1Ea3hOREV4TWpNd01Gb3dQakVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU1V3SXdZRFZRUURFeHh6ZVhOMFpXMDZibTlrWlRweVpXUXRiM05sTFcxaGMzUmxjakF4TUZrd0V3WUgKS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRTNoRmRmN1hiMUN4TFhybG9yOFJhd1h6T2pkWWpYRFRKL045LwpndXpHclBFSngvL25OWU5GaDdocE5PZ0FvaWx4NDA1UWVPUk5qeitBZ2k5VGNpcU41Nk5VTUZJd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWUQKVlIwT0JCWUVGQ2Via0VTMEVEdFpZdEYwQ1FXdkpsbEs4NGF3TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBUQpUbVlpSXdwM1g1LzhCRE11WCtncCttWm1YMUlOZHpmQUVvczhuRGFKNGtUbXVWbTBxckZEYStSUmZLV0kwWVlXCmhmV1ZWbDJXYzB3QzMxMU4za201RTR3L2ptTzk5VmRjdFFTWHFqaC9lREgvTWp2TzZoenpvTUNabWpuV05zK0EKb2NhVXB0UXJ2QkdRb0l6aTlPMVByOFdYYzdmcGpIdjRxRXVGN1dKMEhzMXRUTCtlUXZRbisvTnhJT3pla01NKwpiQ2djMW9BR2U1NXhnN2o4NHYwbElFREdkNVFqWXlpNzlYT2tSZVZxSXJCeUlpTDBtN3BLdE9EeEp1aHpaNDk3CjlVSmk1dVk0WmMwdzFrSFZMWWwxdjVqVWZiQUIrUFMzYy9rNXZOY1krejJ0OCtOOUFxUzBaKzM1YWJpOVFPWUcKSzdJK3ZmV09zcHB4RmRzQitpcE0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", 
>                "conditions": [
>                    {
>                        "lastUpdateTime": "2018-09-14T11:27:57Z", 
>                        "message": "Auto approving kubelet client certificate after SubjectAccessReview.", 
>                        "reason": "AutoApproved", 
>                        "type": "Approved"
>                    }
>                ]
>            }
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:32:27Z", 
>                "generateName": "csr-", 
>                "name": "csr-p8lrx", 
>                "namespace": "", 
>                "resourceVersion": "1900", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-p8lrx", 
>                "uid": "dbcec57e-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:nodes", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVNrdlRFN0tFMFlDY1BzWlZUdG4wMTBmamY3OE5ZcXM2N0llNWlhV3gvYkd1NktBS1ZEQ0g4ZGhhYkEKbjJwN0hxWkxqMFJCWHdYV1BiWUFNTW5jZnY5TW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFPMVhCSEZaSGE0QnNvVjAKMHRVVXNQbHA3RmRqblpEdVpObFhjaVpPYWdQV0FpRUExMStVdnVNbG91T2pyWDdiVWlPNUZKN1dQZm9vay8ydgpHSm9QTjVtMWFPMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:node:red-ose-master01"
>            }, 
>            "status": {}
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:27:50Z", 
>                "generateName": "csr-", 
>                "name": "csr-wsvrj", 
>                "namespace": "", 
>                "resourceVersion": "185", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-wsvrj", 
>                "uid": "36a34d97-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:masters", 
>                    "system:cluster-admins", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlVqQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVJwaXF0VGNDcUVwdVpaM3B3UkpSVVdRZWZXaXVQVmNhYmNEOTl1VVdFY1c3blpjOGh5WTYxUTdmankKNmZNbVExWEhhZGNyY00zbGMrbEU1MFpKVk9sZm9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1h1N2QyOStxU2Y2WUl1c0MKUU4yQ3J4bkIzTUpoWmk2ZkdCMUV4MnJNOWxrQ0lCdE4rNm5QMmN1SnVGZE0zaUlRYitHM0YxaWw3L1lFVHBTTApkcExOd1lmcQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:admin"
>            }, 
>            "status": {}
>        }
>    ]
>}
><red-ose-master01.internal.ose.extrasys.it> ESTABLISH SSH CONNECTION FOR USER: awx
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 red-ose-master01.internal.ose.extrasys.it '/bin/sh -c '"'"'echo ~awx && sleep 0'"'"''
><red-ose-master01.internal.ose.extrasys.it> (0, '/home/awx\n', '')
><red-ose-master01.internal.ose.extrasys.it> ESTABLISH SSH CONNECTION FOR USER: awx
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 red-ose-master01.internal.ose.extrasys.it '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288 `" && echo ansible-tmp-1536924915.55-71944560453288="` echo /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288 `" ) && sleep 0'"'"''
><red-ose-master01.internal.ose.extrasys.it> (0, 'ansible-tmp-1536924915.55-71944560453288=/home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288\n', '')
>Using module file /home/awx/openshift-ansible-openshift-ansible-3.10.47-1/roles/lib_openshift/library/oc_csr_approve.py
><red-ose-master01.internal.ose.extrasys.it> PUT /home/awx/.ansible/tmp/ansible-local-1108rHGAsE/tmp7HcvWb TO /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/oc_csr_approve.py
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 '[red-ose-master01.internal.ose.extrasys.it]'
><red-ose-master01.internal.ose.extrasys.it> (0, 'sftp> put /home/awx/.ansible/tmp/ansible-local-1108rHGAsE/tmp7HcvWb /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/oc_csr_approve.py\n', '')
><red-ose-master01.internal.ose.extrasys.it> ESTABLISH SSH CONNECTION FOR USER: awx
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 red-ose-master01.internal.ose.extrasys.it '/bin/sh -c '"'"'chmod u+x /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/ /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/oc_csr_approve.py && sleep 0'"'"''
><red-ose-master01.internal.ose.extrasys.it> (0, '', '')
><red-ose-master01.internal.ose.extrasys.it> ESTABLISH SSH CONNECTION FOR USER: awx
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 -tt red-ose-master01.internal.ose.extrasys.it '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-zcdegzkzjmywwhefqohmxbxlyshpnkfw; /usr/bin/python /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/oc_csr_approve.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
>Escalation succeeded
><red-ose-master01.internal.ose.extrasys.it> (1, '\r\n{"server_csrs": null, "exception": "  File \\"/tmp/ansible_pJ3nm3/ansible_module_oc_csr_approve.py\\", line 24, in <module>\\n    from json.decoder import JSONDecodeError\\n", "oc_get_nodes": {"items": [{"status": {"capacity": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "8", "memory": "16266348Ki"}, "addresses": [{"type": "InternalIP", "address": "192.168.8.20"}, {"type": "ExternalIP", "address": "156.54.176.240"}, {"type": "Hostname", "address": "red-ose-master01"}], "nodeInfo": {"kernelVersion": "3.10.0-862.11.6.el7.x86_64", "kubeletVersion": "v1.10.0+b81c8f8", "containerRuntimeVersion": "docker://1.13.1", "machineID": "3394c16415a04c30b9c81b1aefb5f404", "kubeProxyVersion": "v1.10.0+b81c8f8", "bootID": "d70c83c3-9154-4c5d-baf3-ed54b2275f28", "osImage": "OpenShift Enterprise", "architecture": "amd64", "systemUUID": "E7777176-96A0-46D0-A6E5-2D409F14AE55", "operatingSystem": "linux"}, "allocatable": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "8", "memory": "16163948Ki"}, "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"sizeBytes": 1268901980, "names": ["registry.access.redhat.com/openshift3/ose-node@sha256:b105ec6800823dc4b1deef0b0fe0abb90afc8e530191606baba014e3d50f1daf", "registry.access.redhat.com/openshift3/ose-node:v3.10"]}, {"sizeBytes": 788612067, "names": ["registry.access.redhat.com/openshift3/ose-control-plane@sha256:3d0b24963b4099bb06e6bf70cd0096c3c332dd814dd6497b60f4ca5902473ca5", "registry.access.redhat.com/openshift3/ose-control-plane:v3.10"]}, {"sizeBytes": 256010737, "names": ["registry.access.redhat.com/rhel7/etcd@sha256:2129d5aa0655c0eb614a94ee5792dca9721e64217509713a6bcbfd2160f9cbfa", "registry.access.redhat.com/rhel7/etcd:3.2.22"]}, {"sizeBytes": 214236553, "names": ["registry.access.redhat.com/openshift3/ose-pod@sha256:6c716eba6a032b5c75690407ef3be8e598047b3b37f3745b71eb67c1a64ee6e0", "registry.access.redhat.com/openshift3/ose-pod:v3.10.34"]}], "conditions": [{"status": "False", "lastTransitionTime": "2018-09-14T11:27:45Z", "reason": "KubeletHasSufficientDisk", "lastHeartbeatTime": "2018-09-14T11:35:09Z", "message": "kubelet has sufficient disk space available", "type": "OutOfDisk"}, {"status": "False", "lastTransitionTime": "2018-09-14T11:27:45Z", "reason": "KubeletHasSufficientMemory", "lastHeartbeatTime": "2018-09-14T11:35:09Z", "message": "kubelet has sufficient memory available", "type": "MemoryPressure"}, {"status": "False", "lastTransitionTime": "2018-09-14T11:27:45Z", "reason": "KubeletHasNoDiskPressure", "lastHeartbeatTime": "2018-09-14T11:35:09Z", "message": "kubelet has no disk pressure", "type": "DiskPressure"}, {"status": "False", "lastTransitionTime": "2018-09-14T11:27:45Z", "reason": "KubeletHasSufficientPID", "lastHeartbeatTime": "2018-09-14T11:35:09Z", "message": "kubelet has sufficient PID available", "type": "PIDPressure"}, {"status": "False", "lastTransitionTime": "2018-09-14T11:27:45Z", "reason": "KubeletNotReady", "lastHeartbeatTime": "2018-09-14T11:35:09Z", "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", "type": "Ready"}]}, "kind": "Node", "spec": {"providerID": "openstack:///e7777176-96a0-46d0-a6e5-2d409f14ae55", "externalID": "e7777176-96a0-46d0-a6e5-2d409f14ae55"}, "apiVersion": "v1", "metadata": {"name": "red-ose-master01", "labels": {"beta.kubernetes.io/os": "linux", "node-role.kubernetes.io/master": "true", "kubernetes.io/hostname": "red-ose-master01", "failure-domain.beta.kubernetes.io/zone": "nova", "beta.kubernetes.io/instance-type": "82369357-d10d-4512-b477-d0f6839bd024", "beta.kubernetes.io/arch": "amd64"}, "namespace": "", "resourceVersion": "2170", "creationTimestamp": "2018-09-14T11:27:47Z", "annotations": {"volumes.kubernetes.io/controller-managed-attach-detach": "true"}, "selfLink": "/api/v1/nodes/red-ose-master01", "uid": "34e7397a-b811-11e8-bb4a-fa163e0c2315"}}], "kind": "List", "apiVersion": "v1", "metadata": {"selfLink": "", "resourceVersion": ""}}, "changed": false, "client_csrs": {"node-csr-o3xS6RXO8DREw1AL2Oeb1ZcDAKZir0KGN6YLd2RlwX8": "red-ose-node01", "node-csr-fHcxz4629_6-JSLhNYx413PrwaRFQPieNwRWR9iJoMw": "red-ose-infra01"}, "failed": true, "state": "unknown", "all_subjects_found": ["subject=/O=system:nodes/CN=system:node:red-ose-master01\\n", "subject=/O=system:nodes/CN=system:node:red-ose-master01\\n", "subject=/O=system:nodes/CN=system:node:red-ose-master01\\n", "subject=/O=system:nodes/CN=system:node:red-ose-master01\\n", "subject=/O=system:nodes/CN=system:node:red-ose-infra01\\n", "subject=/O=system:nodes/CN=system:node:red-ose-node01\\n"], "client_approve_results": [], "rc": 0, "invocation": {"module_args": {"oc_bin": "oc", "oc_conf": "/etc/origin/master/admin.kubeconfig", "node_list": ["red-ose-master01", "red-ose-node01", "red-ose-infra01"]}}, "unwanted_csrs": [{"status": {}, "kind": "CertificateSigningRequest", "spec": {"username": "system:node:red-ose-master01", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVE0dkdnNHZqVFZZdEVGUUJoV1k2cjNHU3NYNGtVTDVYQVN4S1ZlVjJ1RmV6MDV6Q3NVYmtDeTdibmQKOUovYktWSmc1cEhlNitTWW55M24rTlk2eEh4RW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFJbjRBK2JnNXZYQVNHMXcKSjk0UGdwOS9xUU56dnJpWkYxc2tvQ0RYWDNPNUFpRUF5cTE3QXBmekpvRVRLYUx5Sm5pZkFDUFUwajFVSmo3aApCekQ2dGk5cVYzbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", "groups": ["system:nodes", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-mmzmj", "namespace": "", "resourceVersion": "1102", "generateName": "csr-", "creationTimestamp": "2018-09-14T11:30:17Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-mmzmj", "uid": "8e469d12-b811-11e8-bb4a-fa163e0c2315"}}, {"status": {"conditions": [{"message": "Auto approving kubelet client certificate after SubjectAccessReview.", "type": "Approved", "reason": "AutoApproved", "lastUpdateTime": "2018-09-14T11:27:57Z"}], "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNlekNDQVdPZ0F3SUJBZ0lVVVZ2TXgvOUlTb1hpWEhmcTdCU3R2MVZDcVI0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVE0yT1RJME16VXlNQjRYRFRFNApNRGt4TkRFeE1qTXdNRm9YRFRFNU1Ea3hOREV4TWpNd01Gb3dQakVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU1V3SXdZRFZRUURFeHh6ZVhOMFpXMDZibTlrWlRweVpXUXRiM05sTFcxaGMzUmxjakF4TUZrd0V3WUgKS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRTNoRmRmN1hiMUN4TFhybG9yOFJhd1h6T2pkWWpYRFRKL045LwpndXpHclBFSngvL25OWU5GaDdocE5PZ0FvaWx4NDA1UWVPUk5qeitBZ2k5VGNpcU41Nk5VTUZJd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWUQKVlIwT0JCWUVGQ2Via0VTMEVEdFpZdEYwQ1FXdkpsbEs4NGF3TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBUQpUbVlpSXdwM1g1LzhCRE11WCtncCttWm1YMUlOZHpmQUVvczhuRGFKNGtUbXVWbTBxckZEYStSUmZLV0kwWVlXCmhmV1ZWbDJXYzB3QzMxMU4za201RTR3L2ptTzk5VmRjdFFTWHFqaC9lREgvTWp2TzZoenpvTUNabWpuV05zK0EKb2NhVXB0UXJ2QkdRb0l6aTlPMVByOFdYYzdmcGpIdjRxRXVGN1dKMEhzMXRUTCtlUXZRbisvTnhJT3pla01NKwpiQ2djMW9BR2U1NXhnN2o4NHYwbElFREdkNVFqWXlpNzlYT2tSZVZxSXJCeUlpTDBtN3BLdE9EeEp1aHpaNDk3CjlVSmk1dVk0WmMwdzFrSFZMWWwxdjVqVWZiQUIrUFMzYy9rNXZOY1krejJ0OCtOOUFxUzBaKzM1YWJpOVFPWUcKSzdJK3ZmV09zcHB4RmRzQitpcE0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "client auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlINk1JR2dBZ0VBTUQ0eEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVsTUNNR0ExVUVBeE1jYzNsegpkR1Z0T201dlpHVTZjbVZrTFc5elpTMXRZWE4wWlhJd01UQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VICkEwSUFCTjRSWFgrMTI5UXNTMTY1YUsvRVdzRjh6bzNXSTF3MHlmemZmNExzeHF6eENjZi81eldEUlllNGFUVG8KQUtJcGNlTk9VSGprVFk4L2dJSXZVM0lxamVlZ0FEQUtCZ2dxaGtqT1BRUURBZ05KQURCR0FpRUF5RjlsN05DLwpUN2pnMmorQVlLUzVKN2l0ODBrREw5eVo5ZWNxU256SHZuUUNJUURjVVp0dzNrTEJLY2llb3JUcmZBQlFzRUpyCldFdkp4WjU0V3BGYU1idjdOdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-nkgpf", "namespace": "", "resourceVersion": "474", "generateName": "csr-", "creationTimestamp": "2018-09-14T11:27:50Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-nkgpf", "uid": "3651cf91-b811-11e8-bb4a-fa163e0c2315"}}, {"status": {}, "kind": "CertificateSigningRequest", "spec": {"username": "system:node:red-ose-master01", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVNrdlRFN0tFMFlDY1BzWlZUdG4wMTBmamY3OE5ZcXM2N0llNWlhV3gvYkd1NktBS1ZEQ0g4ZGhhYkEKbjJwN0hxWkxqMFJCWHdYV1BiWUFNTW5jZnY5TW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFPMVhCSEZaSGE0QnNvVjAKMHRVVXNQbHA3RmRqblpEdVpObFhjaVpPYWdQV0FpRUExMStVdnVNbG91T2pyWDdiVWlPNUZKN1dQZm9vay8ydgpHSm9QTjVtMWFPMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", "groups": ["system:nodes", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-p8lrx", "namespace": "", "resourceVersion": "1900", "generateName": "csr-", "creationTimestamp": "2018-09-14T11:32:27Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-p8lrx", "uid": "dbcec57e-b811-11e8-bb4a-fa163e0c2315"}}, {"status": {}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlVqQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVJwaXF0VGNDcUVwdVpaM3B3UkpSVVdRZWZXaXVQVmNhYmNEOTl1VVdFY1c3blpjOGh5WTYxUTdmankKNmZNbVExWEhhZGNyY00zbGMrbEU1MFpKVk9sZm9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1h1N2QyOStxU2Y2WUl1c0MKUU4yQ3J4bkIzTUpoWmk2ZkdCMUV4MnJNOWxrQ0lCdE4rNm5QMmN1SnVGZE0zaUlRYitHM0YxaWw3L1lFVHBTTApkcExOd1lmcQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-wsvrj", "namespace": "", "resourceVersion": "185", "generateName": "csr-", "creationTimestamp": "2018-09-14T11:27:50Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-wsvrj", "uid": "36a34d97-b811-11e8-bb4a-fa163e0c2315"}}], "server_approve_results": [], "msg": "Could not find csr for nodes: red-ose-master01"}\r\n', 'Shared connection to red-ose-master01.internal.ose.extrasys.it closed.\r\n')
><red-ose-master01.internal.ose.extrasys.it> ESTABLISH SSH CONNECTION FOR USER: awx
><red-ose-master01.internal.ose.extrasys.it> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=awx -o ConnectTimeout=10 -o ControlPath=/home/awx/.ansible/cp/1d3e57de30 red-ose-master01.internal.ose.extrasys.it '/bin/sh -c '"'"'rm -f -r /home/awx/.ansible/tmp/ansible-tmp-1536924915.55-71944560453288/ > /dev/null 2>&1 && sleep 0'"'"''
><red-ose-master01.internal.ose.extrasys.it> (0, '', '')
>The full traceback is:
>  File "/tmp/ansible_pJ3nm3/ansible_module_oc_csr_approve.py", line 24, in <module>
>    from json.decoder import JSONDecodeError
>
>fatal: [red-ose-master01.internal.ose.extrasys.it]: FAILED! => {
>    "all_subjects_found": [
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-master01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-infra01\n", 
>        "subject=/O=system:nodes/CN=system:node:red-ose-node01\n"
>    ], 
>    "attempts": 30, 
>    "changed": false, 
>    "client_approve_results": [], 
>    "client_csrs": {
>        "node-csr-fHcxz4629_6-JSLhNYx413PrwaRFQPieNwRWR9iJoMw": "red-ose-infra01", 
>        "node-csr-o3xS6RXO8DREw1AL2Oeb1ZcDAKZir0KGN6YLd2RlwX8": "red-ose-node01"
>    }, 
>    "invocation": {
>        "module_args": {
>            "node_list": [
>                "red-ose-master01", 
>                "red-ose-node01", 
>                "red-ose-infra01"
>            ], 
>            "oc_bin": "oc", 
>            "oc_conf": "/etc/origin/master/admin.kubeconfig"
>        }
>    }, 
>    "msg": "Could not find csr for nodes: red-ose-master01", 
>    "oc_get_nodes": {
>        "apiVersion": "v1", 
>        "items": [
>            {
>                "apiVersion": "v1", 
>                "kind": "Node", 
>                "metadata": {
>                    "annotations": {
>                        "volumes.kubernetes.io/controller-managed-attach-detach": "true"
>                    }, 
>                    "creationTimestamp": "2018-09-14T11:27:47Z", 
>                    "labels": {
>                        "beta.kubernetes.io/arch": "amd64", 
>                        "beta.kubernetes.io/instance-type": "82369357-d10d-4512-b477-d0f6839bd024", 
>                        "beta.kubernetes.io/os": "linux", 
>                        "failure-domain.beta.kubernetes.io/zone": "nova", 
>                        "kubernetes.io/hostname": "red-ose-master01", 
>                        "node-role.kubernetes.io/master": "true"
>                    }, 
>                    "name": "red-ose-master01", 
>                    "namespace": "", 
>                    "resourceVersion": "2170", 
>                    "selfLink": "/api/v1/nodes/red-ose-master01", 
>                    "uid": "34e7397a-b811-11e8-bb4a-fa163e0c2315"
>                }, 
>                "spec": {
>                    "externalID": "e7777176-96a0-46d0-a6e5-2d409f14ae55", 
>                    "providerID": "openstack:///e7777176-96a0-46d0-a6e5-2d409f14ae55"
>                }, 
>                "status": {
>                    "addresses": [
>                        {
>                            "address": "192.168.8.20", 
>                            "type": "InternalIP"
>                        }, 
>                        {
>                            "address": "156.54.176.240", 
>                            "type": "ExternalIP"
>                        }, 
>                        {
>                            "address": "red-ose-master01", 
>                            "type": "Hostname"
>                        }
>                    ], 
>                    "allocatable": {
>                        "cpu": "8", 
>                        "hugepages-1Gi": "0", 
>                        "hugepages-2Mi": "0", 
>                        "memory": "16163948Ki", 
>                        "pods": "250"
>                    }, 
>                    "capacity": {
>                        "cpu": "8", 
>                        "hugepages-1Gi": "0", 
>                        "hugepages-2Mi": "0", 
>                        "memory": "16266348Ki", 
>                        "pods": "250"
>                    }, 
>                    "conditions": [
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient disk space available", 
>                            "reason": "KubeletHasSufficientDisk", 
>                            "status": "False", 
>                            "type": "OutOfDisk"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient memory available", 
>                            "reason": "KubeletHasSufficientMemory", 
>                            "status": "False", 
>                            "type": "MemoryPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has no disk pressure", 
>                            "reason": "KubeletHasNoDiskPressure", 
>                            "status": "False", 
>                            "type": "DiskPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "kubelet has sufficient PID available", 
>                            "reason": "KubeletHasSufficientPID", 
>                            "status": "False", 
>                            "type": "PIDPressure"
>                        }, 
>                        {
>                            "lastHeartbeatTime": "2018-09-14T11:35:09Z", 
>                            "lastTransitionTime": "2018-09-14T11:27:45Z", 
>                            "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", 
>                            "reason": "KubeletNotReady", 
>                            "status": "False", 
>                            "type": "Ready"
>                        }
>                    ], 
>                    "daemonEndpoints": {
>                        "kubeletEndpoint": {
>                            "Port": 10250
>                        }
>                    }, 
>                    "images": [
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-node@sha256:b105ec6800823dc4b1deef0b0fe0abb90afc8e530191606baba014e3d50f1daf", 
>                                "registry.access.redhat.com/openshift3/ose-node:v3.10"
>                            ], 
>                            "sizeBytes": 1268901980
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-control-plane@sha256:3d0b24963b4099bb06e6bf70cd0096c3c332dd814dd6497b60f4ca5902473ca5", 
>                                "registry.access.redhat.com/openshift3/ose-control-plane:v3.10"
>                            ], 
>                            "sizeBytes": 788612067
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/rhel7/etcd@sha256:2129d5aa0655c0eb614a94ee5792dca9721e64217509713a6bcbfd2160f9cbfa", 
>                                "registry.access.redhat.com/rhel7/etcd:3.2.22"
>                            ], 
>                            "sizeBytes": 256010737
>                        }, 
>                        {
>                            "names": [
>                                "registry.access.redhat.com/openshift3/ose-pod@sha256:6c716eba6a032b5c75690407ef3be8e598047b3b37f3745b71eb67c1a64ee6e0", 
>                                "registry.access.redhat.com/openshift3/ose-pod:v3.10.34"
>                            ], 
>                            "sizeBytes": 214236553
>                        }
>                    ], 
>                    "nodeInfo": {
>                        "architecture": "amd64", 
>                        "bootID": "d70c83c3-9154-4c5d-baf3-ed54b2275f28", 
>                        "containerRuntimeVersion": "docker://1.13.1", 
>                        "kernelVersion": "3.10.0-862.11.6.el7.x86_64", 
>                        "kubeProxyVersion": "v1.10.0+b81c8f8", 
>                        "kubeletVersion": "v1.10.0+b81c8f8", 
>                        "machineID": "3394c16415a04c30b9c81b1aefb5f404", 
>                        "operatingSystem": "linux", 
>                        "osImage": "OpenShift Enterprise", 
>                        "systemUUID": "E7777176-96A0-46D0-A6E5-2D409F14AE55"
>                    }
>                }
>            }
>        ], 
>        "kind": "List", 
>        "metadata": {
>            "resourceVersion": "", 
>            "selfLink": ""
>        }
>    }, 
>    "rc": 0, 
>    "server_approve_results": [], 
>    "server_csrs": null, 
>    "state": "unknown", 
>    "unwanted_csrs": [
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:30:17Z", 
>                "generateName": "csr-", 
>                "name": "csr-mmzmj", 
>                "namespace": "", 
>                "resourceVersion": "1102", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-mmzmj", 
>                "uid": "8e469d12-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:nodes", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVE0dkdnNHZqVFZZdEVGUUJoV1k2cjNHU3NYNGtVTDVYQVN4S1ZlVjJ1RmV6MDV6Q3NVYmtDeTdibmQKOUovYktWSmc1cEhlNitTWW55M24rTlk2eEh4RW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFJbjRBK2JnNXZYQVNHMXcKSjk0UGdwOS9xUU56dnJpWkYxc2tvQ0RYWDNPNUFpRUF5cTE3QXBmekpvRVRLYUx5Sm5pZkFDUFUwajFVSmo3aApCekQ2dGk5cVYzbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:node:red-ose-master01"
>            }, 
>            "status": {}
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:27:50Z", 
>                "generateName": "csr-", 
>                "name": "csr-nkgpf", 
>                "namespace": "", 
>                "resourceVersion": "474", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-nkgpf", 
>                "uid": "3651cf91-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:masters", 
>                    "system:cluster-admins", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlINk1JR2dBZ0VBTUQ0eEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVsTUNNR0ExVUVBeE1jYzNsegpkR1Z0T201dlpHVTZjbVZrTFc5elpTMXRZWE4wWlhJd01UQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VICkEwSUFCTjRSWFgrMTI5UXNTMTY1YUsvRVdzRjh6bzNXSTF3MHlmemZmNExzeHF6eENjZi81eldEUlllNGFUVG8KQUtJcGNlTk9VSGprVFk4L2dJSXZVM0lxamVlZ0FEQUtCZ2dxaGtqT1BRUURBZ05KQURCR0FpRUF5RjlsN05DLwpUN2pnMmorQVlLUzVKN2l0ODBrREw5eVo5ZWNxU256SHZuUUNJUURjVVp0dzNrTEJLY2llb3JUcmZBQlFzRUpyCldFdkp4WjU0V3BGYU1idjdOdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "client auth"
>                ], 
>                "username": "system:admin"
>            }, 
>            "status": {
>                "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNlekNDQVdPZ0F3SUJBZ0lVVVZ2TXgvOUlTb1hpWEhmcTdCU3R2MVZDcVI0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVE0yT1RJME16VXlNQjRYRFRFNApNRGt4TkRFeE1qTXdNRm9YRFRFNU1Ea3hOREV4TWpNd01Gb3dQakVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU1V3SXdZRFZRUURFeHh6ZVhOMFpXMDZibTlrWlRweVpXUXRiM05sTFcxaGMzUmxjakF4TUZrd0V3WUgKS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRTNoRmRmN1hiMUN4TFhybG9yOFJhd1h6T2pkWWpYRFRKL045LwpndXpHclBFSngvL25OWU5GaDdocE5PZ0FvaWx4NDA1UWVPUk5qeitBZ2k5VGNpcU41Nk5VTUZJd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWUQKVlIwT0JCWUVGQ2Via0VTMEVEdFpZdEYwQ1FXdkpsbEs4NGF3TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBUQpUbVlpSXdwM1g1LzhCRE11WCtncCttWm1YMUlOZHpmQUVvczhuRGFKNGtUbXVWbTBxckZEYStSUmZLV0kwWVlXCmhmV1ZWbDJXYzB3QzMxMU4za201RTR3L2ptTzk5VmRjdFFTWHFqaC9lREgvTWp2TzZoenpvTUNabWpuV05zK0EKb2NhVXB0UXJ2QkdRb0l6aTlPMVByOFdYYzdmcGpIdjRxRXVGN1dKMEhzMXRUTCtlUXZRbisvTnhJT3pla01NKwpiQ2djMW9BR2U1NXhnN2o4NHYwbElFREdkNVFqWXlpNzlYT2tSZVZxSXJCeUlpTDBtN3BLdE9EeEp1aHpaNDk3CjlVSmk1dVk0WmMwdzFrSFZMWWwxdjVqVWZiQUIrUFMzYy9rNXZOY1krejJ0OCtOOUFxUzBaKzM1YWJpOVFPWUcKSzdJK3ZmV09zcHB4RmRzQitpcE0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", 
>                "conditions": [
>                    {
>                        "lastUpdateTime": "2018-09-14T11:27:57Z", 
>                        "message": "Auto approving kubelet client certificate after SubjectAccessReview.", 
>                        "reason": "AutoApproved", 
>                        "type": "Approved"
>                    }
>                ]
>            }
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:32:27Z", 
>                "generateName": "csr-", 
>                "name": "csr-p8lrx", 
>                "namespace": "", 
>                "resourceVersion": "1900", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-p8lrx", 
>                "uid": "dbcec57e-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:nodes", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlZEQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVNrdlRFN0tFMFlDY1BzWlZUdG4wMTBmamY3OE5ZcXM2N0llNWlhV3gvYkd1NktBS1ZEQ0g4ZGhhYkEKbjJwN0hxWkxqMFJCWHdYV1BiWUFNTW5jZnY5TW9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFPMVhCSEZaSGE0QnNvVjAKMHRVVXNQbHA3RmRqblpEdVpObFhjaVpPYWdQV0FpRUExMStVdnVNbG91T2pyWDdiVWlPNUZKN1dQZm9vay8ydgpHSm9QTjVtMWFPMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:node:red-ose-master01"
>            }, 
>            "status": {}
>        }, 
>        {
>            "apiVersion": "certificates.k8s.io/v1beta1", 
>            "kind": "CertificateSigningRequest", 
>            "metadata": {
>                "creationTimestamp": "2018-09-14T11:27:50Z", 
>                "generateName": "csr-", 
>                "name": "csr-wsvrj", 
>                "namespace": "", 
>                "resourceVersion": "185", 
>                "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-wsvrj", 
>                "uid": "36a34d97-b811-11e8-bb4a-fa163e0c2315"
>            }, 
>            "spec": {
>                "groups": [
>                    "system:masters", 
>                    "system:cluster-admins", 
>                    "system:authenticated"
>                ], 
>                "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlVqQ0IrZ0lCQURBK01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14SlRBakJnTlZCQU1USEhONQpjM1JsYlRwdWIyUmxPbkpsWkMxdmMyVXRiV0Z6ZEdWeU1ERXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVJwaXF0VGNDcUVwdVpaM3B3UkpSVVdRZWZXaXVQVmNhYmNEOTl1VVdFY1c3blpjOGh5WTYxUTdmankKNmZNbVExWEhhZGNyY00zbGMrbEU1MFpKVk9sZm9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzd1NUQkhCZ05WSFJFRQpRREErZ2hCeVpXUXRiM05sTFcxaGMzUmxjakF4Z2dDSEJNQ29DQlNIRVA2QUFBQUFBQUFBK0JZKy8vNE1JeFdICkJLd1JBQUdIQk1Db0NCU0hCSncyc1BBd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1h1N2QyOStxU2Y2WUl1c0MKUU4yQ3J4bkIzTUpoWmk2ZkdCMUV4MnJNOWxrQ0lCdE4rNm5QMmN1SnVGZE0zaUlRYitHM0YxaWw3L1lFVHBTTApkcExOd1lmcQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K", 
>                "usages": [
>                    "digital signature", 
>                    "key encipherment", 
>                    "server auth"
>                ], 
>                "username": "system:admin"
>            }, 
>            "status": {}
>        }
>    ]
>}
>	to retry, use: --limit @/home/awx/openshift-ansible-openshift-ansible-3.10.47-1/playbooks/deploy_cluster.retry
>
>PLAY RECAP ********************************************************************************************************************************************************************************
>localhost                  : ok=12   changed=0    unreachable=0    failed=0   
>red-ose-infra01.internal.ose.extrasys.it : ok=125  changed=65   unreachable=0    failed=0   
>red-ose-master01.internal.ose.extrasys.it : ok=449  changed=203  unreachable=0    failed=1   
>red-ose-node01.internal.ose.extrasys.it : ok=125  changed=65   unreachable=0    failed=0   
>
>
>INSTALLER STATUS **************************************************************************************************************************************************************************
>Initialization              : Complete (0:00:39)
>Health Check                : Complete (0:02:51)
>Node Bootstrap Preparation  : Complete (0:14:59)
>etcd Install                : Complete (0:00:55)
>Master Install              : Complete (0:05:36)
>Master Additional Install   : Complete (0:01:36)
>Node Join                   : In Progress (0:02:58)
>	This phase can be restarted by running: playbooks/openshift-node/join.yml
>
>
>Failure summary:
>
>
>  1. Hosts:    red-ose-master01.internal.ose.extrasys.it
>     Play:     Approve any pending CSR requests from inventory nodes
>     Task:     Approve node certificates when bootstrapping
>     Message:  Could not find csr for nodes: red-ose-master01
>

Actions: View

Attachments on bug 1625911: 1481221 | 1481222 | 1481223 | 1481224 | 1483324