Description of problem: A python2-openstacksdk version >= 0.12.0 is required for OpenStack playbooks in OCP 3.11 (ansible 2.6). The 2018-08-22.2 OSP 13 puddle delivers python2-openstackclient-3.14.2-1.el7ost.noarch which is not compatible with OCP 3.11 (ansible 2.6) playbooks. The ansible 2.6 module os_networks_facts fails: fatal: [localhost]: FAILED! => {"changed": false, "msg": "To utilize this module, the installed version ofthe openstacksdk library MUST be >=0.12.0"} Version-Release number of selected component (if applicable): python2-openstackclient-3.14.2-1.el7ost.noarch How reproducible: always Steps to Reproduce: 1. Install ansible 2.6 2. Get OCP 3.11 3. Run Openshift-on-Openstack playbooks: ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml Actual results: TASK [openshift_openstack : Try to get network facts] ************************************************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": false, "msg": "To utilize this module, the installed version ofthe openstacksdk library MUST be >=0.12.0"} Expected results: No errors
provision.yml playbook fails for the same reason, it is the os_stack ansible module in this case: TASK [openshift_openstack : Handle the Stack (create/delete)] ***************************************************************************************************************************************************** FAILED - RETRYING: Handle the Stack (create/delete) (20 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (19 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (18 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (17 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (16 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (15 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (14 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (13 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (12 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (11 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (10 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (9 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (8 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (7 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (6 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (5 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (4 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (3 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (2 retries left). FAILED - RETRYING: Handle the Stack (create/delete) (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 20, "changed": false, "msg": "To utilize this module, the installed version ofthe openstacksdk library MUST be >=0.12.0"} ...ignoring Puddle repo: http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/2018-08-22.2/RH7-RHOS-13.0/x86_64/os/ Current versions: - python2-openstackclient-3.14.2-1.el7ost.noarch - python2-openstacksdk-0.11.3-1.el7ost.noarch Required version: - python2-openstacksdk >=0.12.0
To add, this is true as well for OCP 3.10 if using Ansible 2.6
I'd add a conditional NACK as you are going to be fighting trying to pull in Rocky version openstacksdk into queens. best we can easily get is 0.11.3 which is supported by https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L411 openstacksdk===0.11.3 https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L430 python-openstackclient===3.14.2 if this is required for integration, what all changes are actually required and how is it tested upstream and in RDO.
Regarding the minimal version, I'm afraid that 0.12.0 is a hard requirement from Ansible 2.6: https://github.com/ansible/ansible/blob/341e1a0f64e001bd0b49d91fb4f6712bc4099282/lib/ansible/module_utils/openstack.py#L111-L125 Ansible 2.6 is a hard requirement for openshift-ansible 3.10. Without this change we won't be able to install any version of OpenShift/OCP (and therefore Kuryr) higher than 3.10 on OSP 13 (using the supported packages). Given OSP 13 is an LTS, we feel this is important. Alternatives we've considered: 1. Ship python2-openstacksdk in the Ansible channels - it is an optional Ansible dependency after all - the Ansible team does not ship cloud provider dependencies however 2. Ship it in the OCP channels - similar situation as with Ansible -- they don't ship cloud provider-specific packages 3. Install it from PyPI - ask our users users to install EPEL, then pip and then "pip install openstacksdk" - this is what the OCP folks recommend for installation on AWS which faces similar issues with the boto package - this only needs to happen on the server/VM/laptop running the OpenShift installer (for installation/upgrades/scale out). It's not necessary for the OpenShift nodes themselves - we could do this as a last resort, but we really don't think this a good idea On the other hand, we have shipped shade in part to support the OCP installer which is why we're requesting it here.
(In reply to Tomas Sedovic from comment #12) > Regarding the minimal version, I'm afraid that 0.12.0 is a hard requirement > from Ansible 2.6: > > https://github.com/ansible/ansible/blob/ > 341e1a0f64e001bd0b49d91fb4f6712bc4099282/lib/ansible/module_utils/openstack. > py#L111-L125 > > Ansible 2.6 is a hard requirement for openshift-ansible 3.10. > > Without this change we won't be able to install any version of OpenShift/OCP > (and therefore Kuryr) higher than 3.10 on OSP 13 (using the supported > packages). > > Given OSP 13 is an LTS, we feel this is important. OSP 13 is constrained by upstream openstack upper-constraints[1]. This is currently at 0.11.3. This is also something that doesn't usually get changed after upstream GA. [1] https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L411 > > > Alternatives we've considered: > > 1. Ship python2-openstacksdk in the Ansible channels > - it is an optional Ansible dependency after all > - the Ansible team does not ship cloud provider dependencies however > > 2. Ship it in the OCP channels > - similar situation as with Ansible -- they don't ship cloud > provider-specific packages > > 3. Install it from PyPI > - ask our users users to install EPEL, then pip and then "pip install > openstacksdk" > - this is what the OCP folks recommend for installation on AWS which faces > similar issues with the boto package > - this only needs to happen on the server/VM/laptop running the OpenShift > installer (for installation/upgrades/scale out). It's not necessary for the > OpenShift nodes themselves > - we could do this as a last resort, but we really don't think this a good > idea This isn't even a last resort. PyPi/pip is explicitly not supported by Red Hat. It's roughly equivalent to EPEL. We cannot support content coming from a location other than Red Hat repositories. OSP cannot support this. > > On the other hand, we have shipped shade in part to support the OCP > installer which is why we're requesting it here. We shipped shade for this case, yes, but the version we shipped is the version that works with 13. openstacksdk is depended on by other OSP 13 packages and rebasing can impact those other components. Upstream doesn't explicitly test this combination, so a change in any of the depending projects wouldn't be explicitly tested.
So that was the change introduced with Ansible 2.6 https://github.com/ansible/ansible/commit/89ce826a9fb53c304238923a73667ab820711338 * Switch from shade to openstacksdk IIUC Ansible does NOT actually support openstack module, and if that's so, they should not ship it in the official RPM, instead keep it as separate add-on. @Tomas is openstack module actually used by the OpenShift installer (for installation/upgrades/scale out) and if yes does it use any of the newer feature of the module or it could work with the old version of the module?
Yes, we use the OpenStack Ansible module in the provisioning playbooks (for checking prerequisities are met, creating the Heat stack and getting information about the created resources out). We do not need any features that are new in 2.6. The OpenStack-specific portions should work with the older versions of Ansible (whatever is in OSP 13 is fine). So yes, we could work with an older version of the module. To my knowledge though, the cloud modules are shipped as part of Ansible, not a separate package we could downgrade.
(In reply to Tomas Sedovic from comment #12) > 3. Install it from PyPI > - ask our users users to install EPEL, then pip and then "pip install > openstacksdk" > - this is what the OCP folks recommend for installation on AWS which faces > similar issues with the boto package > - this only needs to happen on the server/VM/laptop running the OpenShift > installer (for installation/upgrades/scale out). It's not necessary for the > OpenShift nodes themselves > - we could do this as a last resort, but we really don't think this a good > idea > > On the other hand, we have shipped shade in part to support the OCP > installer which is why we're requesting it here. As you have said yourself, this is not a good idea - but it is also not a last resort. We simply cannot ask our users to install software from anywhere except Red Hat repositories. If we do it becomes a support/security nightmare. Product Security will block this, if it were to go ahead, so it is best to figure out a better answer. I am not sure what that answer might look like - yet.
(In reply to Jon Schlueter from comment #9) > > https://github.com/openstack/requirements/blob/stable/queens/upper- > constraints.txt#L430 > > python-openstackclient===3.14.2 I think you meant: https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L411 openstacksdk===0.11.3
(Or, additionally, rather)
https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L474 keystoneauth1===3.4.0 https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L297 os-service-types===1.1.0
I've opened a patch to openshift-ansible to allow Ansible 2.5.7. If it gets accepted, we'll backport it. I've tested it upstream on RDO Queens with the same version of packages and it seems to work fine.
Forgot to include the link to the PR: https://github.com/openshift/openshift-ansible/pull/10320
Adding the PR for the backport to 3.11: https://github.com/openshift/openshift-ansible/pull/10472
Please help check if this bug could be verified,thanks!
This change implies using ansible 2.5 for Openstack playbooks, and it works on OSP 13 as it does not require python2-openstacksdk version >= 0.12.0. Trying to run Openstack playbooks with ansible 2.6 will fail. My concern is about the non-openstack Openshift ansible playbooks, are they going to run with ansible 2.5? If the answer is no, different ansible versions are required and need to be documented for different infrastructure platforms. Is this correct?
I believe that as of right now (and at least for 3.11) it should work. Scott please correct me if I'm wrong. But it's not a good long-term solution. Which is why we're trying to get the necessary dependencies shipped in the OCP repos so that we can keep using Ansible 2.6 (and at some point 2.7) and not have a version mismatch like this. I'd say for verifying this particular bug, making sure you can install OCP 3.11 on OSP 13 (with Ansible 2.5) is enough.
There are no known issues with the 3.11 playbooks and Ansible 2.5.x, however that is not something that OpenShift QE has performed complete regression testing on.
Thanks Tomas and Scott. Verified in openshift-ansible-3.11.39-1.git.0.fe42b3b.el7.noarch on top of OSP 13 2018-09-28.1 puddle. python2-openstacksdk version remains the same: python2-openstacksdk-0.11.3-1.el7ost.noarch Ansible 2.5 is required from now on to run OCP 3.11 on OSP 13. Verification steps: 1. Install ansible 2.5 on the bastion instance 2. Get OCP 3.11 3. Run Openshift-on-Openstack playbooks: ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory repos.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory red-hat-ca.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml The playbooks run successfully now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3537
*** Bug 1627008 has been marked as a duplicate of this bug. ***