I'm calling cns-deploy from Ansible using the command module, which just passes the command: cns-deploy -n cns -g /root/cns-topology.json This had been working until some recent changes made it to the stage environment. Now with new deployments I'm seeing the error below (note the TTY message). All nodes are up at this point and the topology file is correct. This works fine if run manually on the master node using the same command and topology file. Output from Ansible run : <snip> "Waiting for GlusterFS pods to start ... OK", "secret \"heketi-config-secret\" created", "secret \"heketi-config-secret\" labeled", "service \"deploy-heketi\" created", "route \"deploy-heketi\" created", "deploymentconfig \"deploy-heketi\" created", "Waiting for deploy-heketi pod to start ... OK", "Unable to use a TTY - input is not a terminal or the right kind of file", "Creating cluster ... ID: 5b048c4edfbc31c3c09fc57e885c9abb", "Allowing file volumes on cluster.", "Allowing block volumes on cluster.", "Creating node cnsreg-cns0.css.lab.eng.bos.redhat.com ... ID: 58b8e66e49fe8e023955797d83f80a5c", "Adding device /dev/vdc ... OK", "Creating node cnsreg-cns1.css.lab.eng.bos.redhat.com ... ID: e966493a9e6103182fa5f162d5ab708b", "Adding device /dev/vdc ... OK", "Creating node cnsreg-cns2.css.lab.eng.bos.redhat.com ... ID: 0d80fe012baa4b3f49184ecce7545e2b", "Adding device /dev/vdc ... OK", "Error loading the cluster topology.", "Please check the failed node or device and rerun this script." Ansible code: - name: Run cns-deploy script command: cns-deploy -n {{ cns_project }} -g /root/cns-topology.json [root@cnsreg-master0 ~]# yum info cns-deploy Loaded plugins: product-id, search-disabled-repos, subscription-manager Installed Packages Name : cns-deploy Arch : x86_64 Version : 5.0.0 Release : 54.el7rhgs Size : 67 k Repo : installed From repo : rh-gluster-3-for-rhel-7-server-rpms Summary : Container-Native Storage deployment tool URL : https://github.com/gluster/gluster-kubernetes License : ASL 2.0 Description : cns-deploy is a tool to facilitate the deployment of a Container-Native Storage : solution using GlusterFS. [root@cnsreg-master0 ~]# oc version oc v3.5.5.31.24 kubernetes v1.5.2+43a9be4 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://cnsreg-master0.css.lab.eng.bos.redhat.com:8443 openshift v3.5.5.31.24 kubernetes v1.5.2+43a9be4 Manual run: <snip> Do you wish to proceed with deployment? [Y]es, [N]o? [Default: Y]: Using OpenShift CLI. Using namespace "cns". Checking for pre-existing resources... GlusterFS pods ... not found. deploy-heketi pod ... not found. heketi pod ... not found. glusterblock-provisioner pod ... not found. gluster-s3 pod ... not found. Creating initial resources ... template "deploy-heketi" created serviceaccount "heketi-service-account" created template "heketi" created template "glusterfs" created role "edit" added: "system:serviceaccount:cns:heketi-service-account" OK node "cnsreg-cns0.css.lab.eng.bos.redhat.com" labeled node "cnsreg-cns1.css.lab.eng.bos.redhat.com" labeled node "cnsreg-cns2.css.lab.eng.bos.redhat.com" labeled daemonset "glusterfs" created Waiting for GlusterFS pods to start ... OK secret "heketi-config-secret" created secret "heketi-config-secret" labeled service "deploy-heketi" created route "deploy-heketi" created deploymentconfig "deploy-heketi" created Waiting for deploy-heketi pod to start ... OK Creating cluster ... ID: 7b0e8d7ca3ff1d0a86dd1a771f870abd Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node cnsreg-cns0.css.lab.eng.bos.redhat.com ... ID: cd969bb58a1344fe637635ae3adf1ceb Adding device /dev/vdc ... OK Creating node cnsreg-cns1.css.lab.eng.bos.redhat.com ... ID: 6feb1e1c8ca4b05f501081ef060bef37 Adding device /dev/vdc ... OK Creating node cnsreg-cns2.css.lab.eng.bos.redhat.com ... ID: 657ec3dfb660be26910939d1f92b273e Adding device /dev/vdc ... OK heketi topology loaded. Saving /tmp/heketi-storage.json secret "heketi-storage-secret" created endpoints "heketi-storage-endpoints" created service "heketi-storage-endpoints" created job "heketi-storage-copy-job" created service "heketi-storage-endpoints" labeled deploymentconfig "deploy-heketi" deleted route "deploy-heketi" deleted service "deploy-heketi" deleted job "heketi-storage-copy-job" deleted pod "deploy-heketi-1-ljjz7" deleted secret "heketi-storage-secret" deleted template "deploy-heketi" deleted service "heketi" created route "heketi" created deploymentconfig "heketi" created Waiting for heketi pod to start ... OK heketi is now running and accessible via http://heketi-cns.css-group.com . To run administrative commands you can install 'heketi-cli' and use it as follows: # heketi-cli -s http://heketi-cns.css-group.com --user admin --secret '<ADMIN_KEY>' cluster list You can find it at https://github.com/heketi/heketi/releases . Alternatively, use it from within the heketi pod: # /usr/bin/oc -n cns exec -it <HEKETI_POD> -- heketi-cli -s http://localhost:8080 --user admin --secret '<ADMIN_KEY>' cluster list For dynamic provisioning, create a StorageClass similar to this: --- apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: glusterfs-storage provisioner: kubernetes.io/glusterfs parameters: resturl: "http://heketi-cns.css-group.com" Ready to create and provide GlusterFS volumes. clusterrole "glusterblock-provisioner-runner" created serviceaccount "glusterblock-provisioner" created clusterrolebinding "glusterblock-provisioner" created deploymentconfig "glusterblock-provisioner-dc" created Waiting for glusterblock-provisioner pod to start ... OK Ready to create and provide Gluster block volumes. Deployment complete! [root@cnsreg-master0 ~]# Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
Adding topology file: [root@cnsreg-master0 ~]# cat /root/cns-topology.json { "clusters": [ { "nodes": [ { "node": { "hostnames": { "manage": [ "cnsreg-cns0.css.lab.eng.bos.redhat.com" ], "storage": [ "192.168.1.52" ] }, "zone": 1 }, "devices": [ "/dev/vdc" ] }, { "node": { "hostnames": { "manage": [ "cnsreg-cns1.css.lab.eng.bos.redhat.com" ], "storage": [ "192.168.1.61" ] }, "zone": 1 }, "devices": [ "/dev/vdc" ] }, { "node": { "hostnames": { "manage": [ "cnsreg-cns2.css.lab.eng.bos.redhat.com" ], "storage": [ "192.168.1.72" ] }, "zone": 1 }, "devices": [ "/dev/vdc" ] } ] } ] } [root@cnsreg-master0 ~]#
I hit the same issue via ssh also - Unable to use a TTY - input is not a terminal or the right kind of file. [root@dhcp35-120 cns-libs]# ssh root.47.125 cns-deploy -n storage-project -g -c oc -t /usr/share/heketi/templates -l cns_deploy.log -v -w 600 -y /usr/share/heketi/topology.json Using OpenShift CLI. Checking status of namespace matching 'storage-project': storage-project Active 2d Using namespace "storage-project". Checking for pre-existing resources... GlusterFS pods ... Checking status of pods matching '--selector=glusterfs=pod': No resources found. Timed out waiting for pods matching '--selector=glusterfs=pod'. not found. deploy-heketi pod ... Checking status of pods matching '--selector=deploy-heketi=pod': No resources found. Timed out waiting for pods matching '--selector=deploy-heketi=pod'. not found. heketi pod ... Checking status of pods matching '--selector=heketi=pod': No resources found. Timed out waiting for pods matching '--selector=heketi=pod'. not found. glusterblock-provisioner pod ... Checking status of pods matching '--selector=glusterfs=block-provisioner-pod': No resources found. Timed out waiting for pods matching '--selector=glusterfs=block-provisioner-pod'. not found. gluster-s3 pod ... Checking status of pods matching '--selector=glusterfs=s3-pod': No resources found. Timed out waiting for pods matching '--selector=glusterfs=s3-pod'. not found. Creating initial resources ... oc -n storage-project create -f /usr/share/heketi/templates/deploy-heketi-template.yaml 2>&1 template "deploy-heketi" created oc -n storage-project create -f /usr/share/heketi/templates/heketi-service-account.yaml 2>&1 serviceaccount "heketi-service-account" created oc -n storage-project create -f /usr/share/heketi/templates/heketi-template.yaml 2>&1 template "heketi" created oc -n storage-project create -f /usr/share/heketi/templates/glusterfs-template.yaml 2>&1 template "glusterfs" created oc -n storage-project policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account 2>&1 role "edit" added: "system:serviceaccount:storage-project:heketi-service-account" oc -n storage-project adm policy add-scc-to-user privileged -z heketi-service-account OK Marking 'apek-app-0' as a GlusterFS node. oc -n storage-project label nodes apek-app-0 storagenode=glusterfs 2>&1 node "apek-app-0" labeled Marking 'apek-app-1' as a GlusterFS node. oc -n storage-project label nodes apek-app-1 storagenode=glusterfs 2>&1 node "apek-app-1" labeled Marking 'apek-app-2' as a GlusterFS node. oc -n storage-project label nodes apek-app-2 storagenode=glusterfs 2>&1 node "apek-app-2" labeled Deploying GlusterFS pods. oc -n storage-project process -p NODE_LABEL=glusterfs glusterfs | oc -n storage-project create -f - 2>&1 daemonset "glusterfs" created Waiting for GlusterFS pods to start ... Checking status of pods matching '--selector=glusterfs=pod': glusterfs-lfdkb 1/1 Running 0 1m glusterfs-sd162 1/1 Running 0 1m glusterfs-w663w 1/1 Running 0 1m OK oc -n storage-project create secret generic heketi-config-secret --from-file=private_key=/dev/null --from-file=./heketi.json --from-file=topology.json=/usr/share/heketi/topology.json secret "heketi-config-secret" created oc -n storage-project label --overwrite secret heketi-config-secret glusterfs=heketi-config-secret heketi=config-secret secret "heketi-config-secret" labeled oc -n storage-project process -p HEKETI_EXECUTOR=kubernetes -p HEKETI_FSTAB=/var/lib/heketi/fstab -p HEKETI_ADMIN_KEY= -p HEKETI_USER_KEY= deploy-heketi | oc -n storage-project create -f - 2>&1 service "deploy-heketi" created route "deploy-heketi" created deploymentconfig "deploy-heketi" created Waiting for deploy-heketi pod to start ... Checking status of pods matching '--selector=deploy-heketi=pod': deploy-heketi-1-drh9n 1/1 Running 0 13s OK Determining heketi service URL ... OK oc -n storage-project exec -it deploy-heketi-1-drh9n -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1 Unable to use a TTY - input is not a terminal or the right kind of file Creating cluster ... ID: 0870f7df0b37f9e6944ebace60165ed0 Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node apek-app-0 ... ID: 0eb64f6ddcf02f9ab05653107753e862 Adding device /dev/sdd ... OK Adding device /dev/sde ... OK Creating node apek-app-1 ... ID: 5bdfdafcfd0f90138a7f21884de3b5e0 Adding device /dev/sdd ... OK Adding device /dev/sde ... OK Creating node apek-app-2 ... ID: 3166250234f998d26688a9afa7a9254e Adding device /dev/sdd ... OK Adding device /dev/sde ... OK Error loading the cluster topology. Please check the failed node or device and rerun this script. The automation runs are failling because of this. Note that this command used to work before with 3.6 builds(its failing from approx 2 weeks).
cns-deploy version - cns-deploy-5.0.0-59.el7rhgs.x86_64
This seems to be fixed upstream with https://github.com/gluster/gluster-kubernetes/pull/329
Thanks, I am flagging this for CNS 3.9 as its a regression.
This is fixed in latest>=cns-deploy-6.0.0-2.el7rhgs
Works fine on build - cns-deploy-6.0.0-5.el7rhgs.x86_64
Is it possible to get a 3.7 backport of this ?
Is it possible to get a 3.6 backport of this? - case attached on BZ.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0645