Description of problem: openshift on openstack does not work with valid clouds.yaml files if those do not contain "project_id": This file here works and creates workers: ~~~ clouds: overcloud: auth: auth_url: http://172.16.0.130:5000//v3 username: "admin" password: zP4be2ukhpCkj4zqRfUk8XjQb project_id: 1bb14f515f0945a4891fe3fa2372a795 project_name: "admin" user_domain_name: "Default" region_name: "regionOne" interface: "public" identity_api_version: 3 ~~~ This here doesn't: ~~~ clouds: overcloud: auth: auth_url: http://172.16.0.130:5000//v3 username: "admin" password: zP4be2ukhpCkj4zqRfUk8XjQb project_name: "admin" user_domain_name: "Default" region_name: "regionOne" interface: "public" identity_api_version: 3 ~~~ However, both clouds.yaml files are equally valid. You can run each against the cloud and it will work. The project_id is not necessary, and it's also not provided in file `overcloudrc`. The example without project_id fails on worker creation with: ~~~ 2019-12-20T10:23:14.635223358+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName ~~~ Troubleshooting steps: ~~~ (overcloud) [stack@undercloud-0 clouds]$ oc describe machine -n openshift-machine-api osc-c5r5c-worker-bgt9b Name: osc-c5r5c-worker-bgt9b Namespace: openshift-machine-api Labels: machine.openshift.io/cluster-api-cluster=osc-c5r5c machine.openshift.io/cluster-api-machine-role=worker machine.openshift.io/cluster-api-machine-type=worker machine.openshift.io/cluster-api-machineset=osc-c5r5c-worker Annotations: <none> API Version: machine.openshift.io/v1beta1 Kind: Machine Metadata: Creation Timestamp: 2019-12-20T10:23:14Z Finalizers: machine.machine.openshift.io Generate Name: osc-c5r5c-worker- Generation: 1 Owner References: API Version: machine.openshift.io/v1beta1 Block Owner Deletion: true Controller: true Kind: MachineSet Name: osc-c5r5c-worker UID: 9077b6df-2312-11ea-9b6c-fa163e431263 Resource Version: 3455 Self Link: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/osc-c5r5c-worker-bgt9b UID: bb0af782-2312-11ea-9b6c-fa163e431263 Spec: Metadata: Creation Timestamp: <nil> Provider Spec: Value: API Version: openstackproviderconfig.openshift.io/v1alpha1 Cloud Name: openstack Clouds Secret: Name: openstack-cloud-credentials Namespace: openshift-machine-api Flavor: m1.openshift Image: rhcos Kind: OpenstackProviderSpec Metadata: Creation Timestamp: <nil> Networks: Filter: Subnets: Filter: Name: osc-c5r5c-nodes Tags: openshiftClusterID=osc-c5r5c Security Groups: Filter: Name: osc-c5r5c-worker Server Metadata: Name: osc-c5r5c-worker Openshift Cluster ID: osc-c5r5c Tags: openshiftClusterID=osc-c5r5c Trunk: true User Data Secret: Name: worker-user-data Events: <none> (overcloud) [stack@undercloud-0 clouds]$ oc get machine -n openshift-machine-api NAME STATE TYPE REGION ZONE AGE osc-c5r5c-master-0 4h52m osc-c5r5c-master-1 4h52m osc-c5r5c-master-2 4h52m osc-c5r5c-worker-bgt9b 4h51m osc-c5r5c-worker-qphk7 4h51m osc-c5r5c-worker-vs85h 4h51m (overcloud) [stack@undercloud-0 clouds]$ oc get machineset -n openshift-machine-api NAME DESIRED CURRENT READY AVAILABLE AGE osc-c5r5c-worker 3 3 4h54m (overcloud) [stack@undercloud-0 clouds]$ ~~~ ~~~ (overcloud) [stack@undercloud-0 clouds]$ kubectl get machineset -n openshift-machine-api osc-c5r5c-worker -o yaml apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: creationTimestamp: "2019-12-20T10:22:02Z" generation: 1 labels: machine.openshift.io/cluster-api-cluster: osc-c5r5c machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker name: osc-c5r5c-worker namespace: openshift-machine-api resourceVersion: "3448" selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machinesets/osc-c5r5c-worker uid: 9077b6df-2312-11ea-9b6c-fa163e431263 spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-cluster: osc-c5r5c machine.openshift.io/cluster-api-machineset: osc-c5r5c-worker template: metadata: creationTimestamp: null labels: machine.openshift.io/cluster-api-cluster: osc-c5r5c machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: osc-c5r5c-worker spec: metadata: creationTimestamp: null providerSpec: value: apiVersion: openstackproviderconfig.openshift.io/v1alpha1 cloudName: openstack cloudsSecret: name: openstack-cloud-credentials namespace: openshift-machine-api flavor: m1.openshift image: rhcos kind: OpenstackProviderSpec metadata: creationTimestamp: null networks: - filter: {} subnets: - filter: name: osc-c5r5c-nodes tags: openshiftClusterID=osc-c5r5c securityGroups: - filter: {} name: osc-c5r5c-worker serverMetadata: Name: osc-c5r5c-worker openshiftClusterID: osc-c5r5c tags: - openshiftClusterID=osc-c5r5c trunk: true userDataSecret: name: worker-user-data status: fullyLabeledReplicas: 3 observedGeneration: 1 replicas: 3 ~~~ On master node 0: ~~~ [stack@undercloud-0 ~]$ eval $(ssh-agent) [stack@undercloud-0 ~]$ ssh-add ~/.ssh/id_rsa [stack@undercloud-0 ~]$ ssh core.redhat.local -A [core@o(...)master-1 ~]$ ssh core@(...)master-0 ~~~ ~~~ /var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-tm7qc_b1f05af2-2312-11ea-9b6c-fa163e431263/machine-controller/0.log (...) 2019-12-20T10:23:14.635223358+00:00 stderr F E1220 10:23:14.635191 1 controller.go:239] Failed to check if machine "osc-c5r5c-worker-bgt9b" exists: Error checking if instance exists (machine/actuator.go 346): 2019-12-20T10:23:14.635223358+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName 2019-12-20T10:23:15.636135623+00:00 stderr F I1220 10:23:15.635874 1 controller.go:133] Reconciling Machine "osc-c5r5c-master-0" 2019-12-20T10:23:15.636215058+00:00 stderr F I1220 10:23:15.636192 1 controller.go:304] Machine "osc-c5r5c-master-0" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster 2019-12-20T10:23:15.641348382+00:00 stderr F E1220 10:23:15.641317 1 controller.go:239] Failed to check if machine "osc-c5r5c-master-0" exists: Error checking if instance exists (machine/actuator.go 346): 2019-12-20T10:23:15.641348382+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName 2019-12-20T10:23:16.641614350+00:00 stderr F I1220 10:23:16.641562 1 controller.go:133] Reconciling Machine "osc-c5r5c-master-1" 2019-12-20T10:23:16.641614350+00:00 stderr F I1220 10:23:16.641591 1 controller.go:304] Machine "osc-c5r5c-master-1" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster 2019-12-20T10:23:16.647323379+00:00 stderr F E1220 10:23:16.647295 1 controller.go:239] Failed to check if machine "osc-c5r5c-master-1" exists: Error checking if instance exists (machine/actuator.go 346): 2019-12-20T10:23:16.647323379+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName 2019-12-20T10:23:17.647630531+00:00 stderr F I1220 10:23:17.647587 1 controller.go:133] Reconciling Machine "osc-c5r5c-master-2" 2019-12-20T10:23:17.647695173+00:00 stderr F I1220 10:23:17.647677 1 controller.go:304] Machine "osc-c5r5c-master-2" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster 2019-12-20T10:23:17.652613228+00:00 stderr F E1220 10:23:17.652551 1 controller.go:239] Failed to check if machine "osc-c5r5c-master-2" exists: Error checking if instance exists (machine/actuator.go 346): 2019-12-20T10:23:17.652613228+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName 2019-12-20T10:23:18.652945792+00:00 stderr F I1220 10:23:18.652884 1 controller.go:133] Reconciling Machine "osc-c5r5c-worker-qphk7" ~~~ Looking at the secret: ~~~ (overcloud) [stack@undercloud-0 clouds]$ kubectl get secrets -n openshift-machine-api openstack-cloud-credentials -o yaml apiVersion: v1 data: clouds.yaml: Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogIiIKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg== kind: Secret metadata: annotations: cloudcredential.openshift.io/credentials-request: openshift-cloud-credential-operator/openshift-machine-api-openstack creationTimestamp: "2019-12-20T10:23:00Z" name: openstack-cloud-credentials namespace: openshift-machine-api resourceVersion: "2525" selfLink: /api/v1/namespaces/openshift-machine-api/secrets/openstack-cloud-credentials uid: b2bded52-2312-11ea-9b6c-fa163e431263 type: Opaque ~~~ ~~~ (overcloud) [stack@undercloud-0 clouds]$ base64 -d <(echo 'Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogIiIKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg==') clouds: openstack: auth: application_credential_id: "" application_credential_name: "" application_credential_secret: "" auth_url: http://172.16.0.130:5000//v3 default_domain: "" domain_id: "" domain_name: "" password: zP4be2ukhpCkj4zqRfUk8XjQb project_domain_id: "" project_domain_name: "" project_id: "" project_name: admin token: "" user_domain_id: "" user_domain_name: Default user_id: "" username: admin auth_type: "" cacert: "" cert: "" cloud: "" identity_api_version: "3" key: "" profile: "" region_name: regionOne regions: null verify: true volume_api_version: "" ~~~ We can see that the above clouds.yaml is generated with way more parameters than the original files. And these parameters are left empty. And I think that this is the reason why this fails. Although I don't have time to troubleshoot this further. The example with project_id will work, however: ~~~ [stack@undercloud-0 clouds]$ export KUBECONFIG=install-config/auth/kubeconfig [stack@undercloud-0 clouds]$ oc get machineset -A NAMESPACE NAME DESIRED CURRENT READY AVAILABLE AGE openshift-machine-api osc-kvbd5-worker 3 3 4m33s [stack@undercloud-0 clouds]$ oc get machines -A NAMESPACE NAME STATE TYPE REGION ZONE AGE openshift-machine-api osc-kvbd5-master-0 ACTIVE m1.openshift regionOne nova 4m35s openshift-machine-api osc-kvbd5-master-1 ACTIVE m1.openshift regionOne nova 4m35s openshift-machine-api osc-kvbd5-master-2 ACTIVE m1.openshift regionOne nova 4m35s openshift-machine-api osc-kvbd5-worker-984jk ACTIVE m1.openshift regionOne nova 3m4s openshift-machine-api osc-kvbd5-worker-cvmpg ACTIVE m1.openshift regionOne nova 3m4s openshift-machine-api osc-kvbd5-worker-s4nk8 ACTIVE m1.openshift regionOne nova 3m4s [stack@undercloud-0 clouds]$ kubectl get secrets -n openshift-machine-api openstack-cloud-credentials -o yaml apiVersion: v1 data: clouds.yaml: Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogMWJiMTRmNTE1ZjA5NDVhNDg5MWZlM2ZhMjM3MmE3OTUKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg== kind: Secret metadata: annotations: cloudcredential.openshift.io/credentials-request: openshift-cloud-credential-operator/openshift-machine-api-openstack creationTimestamp: "2019-12-20T18:02:32Z" name: openstack-cloud-credentials namespace: openshift-machine-api resourceVersion: "2894" selfLink: /api/v1/namespaces/openshift-machine-api/secrets/openstack-cloud-credentials uid: e4f56c69-2352-11ea-a185-fa163e5172ff type: Opaque [stack@undercloud-0 clouds]$ base64 -d <(echo 'Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogMWJiMTRmNTE1ZjA5NDVhNDg5MWZlM2ZhMjM3MmE3OTUKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg==') clouds: openstack: auth: application_credential_id: "" application_credential_name: "" application_credential_secret: "" auth_url: http://172.16.0.130:5000//v3 default_domain: "" domain_id: "" domain_name: "" password: zP4be2ukhpCkj4zqRfUk8XjQb project_domain_id: "" project_domain_name: "" project_id: 1bb14f515f0945a4891fe3fa2372a795 project_name: admin token: "" user_domain_id: "" user_domain_name: Default user_id: "" username: admin auth_type: "" cacert: "" cert: "" cloud: "" identity_api_version: "3" key: "" profile: "" region_name: regionOne regions: null verify: true volume_api_version: "" ~~~ Version-Release number of the following components: ~~~ [stack@undercloud-0 clouds]$ rpm -q openshift-ansible package openshift-ansible is not installed [stack@undercloud-0 clouds]$ rpm -q ansible ansible-2.6.19-1.el7ae.noarch [stack@undercloud-0 clouds]$ ansible --version ansible 2.6.19 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/stack/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] [stack@undercloud-0 clouds]$ [stack@undercloud-0 clouds]$ ~~~ How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Setting target to the active development branch (4.4). Clones of this BZ will be created for fixes, if any, which are required to be backported to earlier release maintenance streams.
This was fixed in 4.3 with https://github.com/openshift/installer/pull/2726. Marking this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1776272. *** This bug has been marked as a duplicate of bug 1776272 ***
Fixed in 4.4, sorry.