Description of problem: While running openstack overcloud ffwd-upgrade run, the overcloud nodes fail when installing openstack-heat-agents Version-Release number of selected component (if applicable): 10->13 How reproducible: 100% Steps to Reproduce: 1. Go through ffwd upgrade process up to run 2. Execute ffwd-upgrade run (using CDN subscriptions and repos) 3. Failure occurs while installing latest openstack-heat-agents Actual results: Failure during openstack overcloud upgrade run Expected results: Successful completion of that steps. Additional info: [root@lab-ceph01 ~]# yum install openstack-heat-agents ... --> Finished Dependency Resolution Error: Package: ansible-2.4.6.0-1.el7ae.noarch (rhel-7-server-openstack-13-rpms) Requires: sshpass Error: Package: python-paunch-2.5.0-3.el7ost.noarch (rhel-7-server-openstack-13-rpms) Requires: docker ********************************************************************** yum can be configured to try to resolve such errors by temporarily enabling disabled repos and searching for missing dependencies. To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf ********************************************************************** Error: Package: ansible-2.4.6.0-1.el7ae.noarch (rhel-7-server-openstack-13-rpms) Requires: sshpass Error: Package: python-paunch-2.5.0-3.el7ost.noarch (rhel-7-server-openstack-13-rpms) Requires: docker You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
[u'fatal: [lab-ceph03]: FAILED! => {"changed": true, "failed": true, "msg": "Error: Package: ansible-2.4.6.0-1.el7ae.noarch (rhel-7-server-openstack-13-rpms)\\n Requires: sshpass\\nError: Package: python-paunch-2.5.0-3.el7ost.noarch (rhel-7-server-openstack-13-rpms)\\n Requires: docker\\n", "rc": 1, "results": ["Loaded plugins: product-id, search-disabled-repos, subscription-manager\\nResolving Dependencies\\n--> Running transaction check\\n---> Package openstack-heat-agents.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n--> Processing Dependency: python-heat-agent = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-ansible = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-apply-config = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-docker-cmd = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-hiera = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-json-file = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Processing Dependency: python-heat-agent-puppet = 1.5.4-0.20180308153305.ecf43c7.el7ost for package: openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n--> Running transaction check\\n---> Package python-heat-agent.noarch 0:0-0.15.1e6015dgit.el7ost will be updated\\n---> Package python-heat-agent.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be an update\\n---> Package python-heat-agent-ansible.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n--> Processing Dependency: ansible for package: python-heat-agent-ansible-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n---> Package python-heat-agent-apply-config.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n---> Package python-heat-agent-docker-cmd.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n--> Processing Dependency: python-paunch for package: python-heat-agent-docker-cmd-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch\\n---> Package python-heat-agent-hiera.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n---> Package python-heat-agent-json-file.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be installed\\n---> Package python-heat-agent-puppet.noarch 0:0-0.15.1e6015dgit.el7ost will be updated\\n---> Package python-heat-agent-puppet.noarch 0:1.5.4-0.20180308153305.ecf43c7.el7ost will be an update\\n--> Running transaction check\\n---> Package ansible.noarch 0:2.4.6.0-1.el7ae will be installed\\n--> Processing Dependency: python-jmespath for package: ansible-2.4.6.0-1.el7ae.noarch\\n--> Processing Dependency: sshpass for package: ansible-2.4.6.0-1.el7ae.noarch\\n---> Package python-paunch.noarch 0:2.5.0-3.el7ost will be installed\\n--> Processing Dependency: python2-tenacity >= 3.2.1 for package: python-paunch-2.5.0-3.el7ost.noarch\\n--> Processing Dependency: docker for package: python-paunch-2.5.0-3.el7ost.noarch\\n--> Running transaction check\\n---> Package ansible.noarch 0:2.4.6.0-1.el7ae will be installed\\n--> Processing Dependency: sshpass for package: ansible-2.4.6.0-1.el7ae.noarch\\n---> Package python-jmespath.noarch 0:0.9.0-5.el7ost will be installed\\n---> Package python-paunch.noarch 0:2.5.0-3.el7ost will be installed\\n--> Processing Dependency: docker for package: python-paunch-2.5.0-3.el7ost.noarch\\n---> Package python2-tenacity.noarch 0:4.8.0-1.el7ost will be installed\\n--> Finished Dependency Resolution\\nError: Package: ansible-2.4.6.0-1.el7ae.noarch (rhel-7-server-openstack-13-rpms)\\n Requires: sshpass\\nError: Package: python-paunch-2.5.0-3.el7ost.noarch (rhel-7-server-openstack-13-rpms)\\n Requires: docker\\n**********************************************************************\\nyum can be configured to try to resolve such errors by temporarily enabling\\ndisabled repos and searching for missing dependencies.\\nTo enable this functionality please set \'notify_only=0\' in /etc/yum/pluginconf.d/search-disabled-repos.conf\\n**********************************************************************\\n\\n You could try using --skip-broken to work around the problem\\n You could try running: rpm -Va --nofiles --nodigest\\n"]}',
Created attachment 1528170 [details] package_update log from ffwd-upgrade run
What version of RHEL was used and what exact version of OSP13 was used? It's either missing rhel-7-server-extras or skipped update to RHEL7.6 on OSP10.
(undercloud) [stack@undercloud ~]$ openstack server list +--------------------------------------+------------------+--------+----------------------+--------------------------------+--------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------+--------+----------------------+--------------------------------+--------------+ | 59da6272-8994-4afc-ba73-3ef63421e1e1 | lab-controller03 | ACTIVE | ctlplane=172.16.0.32 | overcloud-full_20190205T220525 | baremetal | | e79f0dc0-2e5f-4639-92f6-f6405beddb6d | lab-controller01 | ACTIVE | ctlplane=172.16.0.36 | overcloud-full_20190205T220525 | baremetal | | 2e759aa4-e37f-48de-b474-671e04e68c0c | lab-controller02 | ACTIVE | ctlplane=172.16.0.25 | overcloud-full_20190205T220525 | baremetal | | 18ee06c0-840c-4484-b969-1cb4c31f7363 | lab-compute01 | ACTIVE | ctlplane=172.16.0.27 | overcloud-full_20190205T220525 | compute | | c678c711-6cc5-4837-a13f-90c60e664901 | lab-ceph03 | ACTIVE | ctlplane=172.16.0.29 | overcloud-full_20190205T220525 | ceph-storage | | f1e8f988-dbfd-43f4-ae04-e260d5e6c0bc | lab-compute02 | ACTIVE | ctlplane=172.16.0.31 | overcloud-full_20190205T220525 | compute | | 3fe27817-3535-41ce-ab79-39fe02b0a968 | lab-ceph01 | ACTIVE | ctlplane=172.16.0.30 | overcloud-full_20190205T220525 | ceph-storage | | 6486e06e-afd0-4c5a-8de0-e02e5ce70f35 | lab-ceph02 | ACTIVE | ctlplane=172.16.0.21 | overcloud-full_20190205T220525 | ceph-storage | +--------------------------------------+------------------+--------+----------------------+--------------------------------+--------------+ Used the default ffu_update_repo.sh in the upgrade. [root@lab-controller03 ~]# cat ffu_update_repo.sh #!/bin/bash set -e echo "If you use FastForwardRepoType 'custom-script' you have to provide the upgrade repo script content." echo "It will be installed as /root/ffu_upgrade_repo.sh on the node" echo "and passed the upstream name (ocata, pike, queens) of the release as first argument" case $1 in ocata) subscription-manager repos --disable=rhel-7-server-openstack-10-rpms subscription-manager repos --enable=rhel-7-server-openstack-11-rpms ;; pike) subscription-manager repos --disable=rhel-7-server-openstack-11-rpms subscription-manager repos --enable=rhel-7-server-openstack-12-rpms ;; queens) subscription-manager repos --disable=rhel-7-server-openstack-12-rpms subscription-manager repos --enable=rhel-7-server-openstack-13-rpms subscription-manager repos --disable=rhel-7-server-rhceph-2-osd-rpms || true subscription-manager repos --disable=rhel-7-server-rhceph-2-mon-rpms subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms subscription-manager repos --disable=rhel-7-server-rhceph-2-tools-rpms subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms ;; *) echo "unknown release $1" >&2 exit 1 esac Current release and kernel version (undercloud) [stack@undercloud ~]$ for i in {32,36,25,27,29,31,30,21}; do ssh heat-admin.0.$i cat /etc/redhat-release; done Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) Red Hat Enterprise Linux Server release 7.6 (Maipo) (undercloud) [stack@undercloud ~]$ for i in {32,36,25,27,29,31,30,21}; do ssh heat-admin.0.$i sudo uname -a; done Linux lab-controller03 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-controller01 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-controller02 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-compute01 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-ceph03 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-compute02 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-ceph01 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Linux lab-ceph02 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Here is the original repo enable command that was run when they were OSP 10 (pre upgrade) for i in {32,36,25,27,29,31,30,21}; do ssh heat-admin.0.$i sudo subscription-manager repos --disable=* --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms --enable=rhel-7-server-satellite-tools-6.3-rpms --enable=rhel-7-server-rhceph-2-osd-rpms --enable=rhel-7-server-rhceph-2-mon-rpms --enable=rhel-7-server-rhceph-2-tools-rpms; done Here is the repo list on each overcloud node after upgrade process failed. Looks like something definitely got missed when updating the enabled repos. (undercloud) [stack@undercloud ~]$ for i in {32,36,25,27,29,31,30,21}; do ssh heat-admin.0.$i sudo yum repolist; done Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-12-rpms/x86_64 Red Hat OpenStack Platform 12 for 1,457 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server 23,675 repolist: 25,132 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751 Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status !rhel-7-server-openstack-13-rpms/x86_64 Red Hat OpenStack Platform 13 f 1,541 !rhel-7-server-rhceph-3-mon-rpms/x86_64 Red Hat Ceph Storage MON 3 for 220 !rhel-7-server-rhceph-3-tools-rpms/x86_64 Red Hat Ceph Storage Tools 3 fo 315 !rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Serv 23,675 repolist: 25,751
I don't see rhel-7-server-extras-rpms enabled in listed repos.
Correct. That's the point. Somewhere along the way, repos got disabled that were enabled. Unfortunately I don't know when that happened. You can see in my original command where it was enabled when the cluster was OSP 10. for i in {32,36,25,27,29,31,30,21}; do ssh heat-admin.0.$i sudo subscription-manager repos --disable=* --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms --enable=rhel-7-server-satellite-tools-6.3-rpms --enable=rhel-7-server-rhceph-2-osd-rpms --enable=rhel-7-server-rhceph-2-mon-rpms --enable=rhel-7-server-rhceph-2-tools-rpms; done
We need logs from the nodes as we need to know when subscription-manager removed the repo. Otherwise we can close as works for me as we didn't reproduce this.
We were not able to reproduce this.
I've just hit exactly the same situation when working through the FFU process by hand using an environment registered with subscription-manager. The only exception to the above information is that I have RHEL 7.7 present. I'm looking through logs right now to try and identify what disables the extras repository.
In my case, I had not done the following instructions from [1]: "you need to remove the old rhel-registration environment files from your deployment command, eg: environment-rhel-registration.yaml; rhel-registration-resource-registry.yaml" I have confirmed that a forced re-registration of the nodes with CDN disables all the required repositories. I'm busy verifying whether removing the rhel-registration environment files prior to executing the ffwd-upgrade resolves the issue for my environment, at least. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/fast_forward_upgrades/assembly-preparing_for_overcloud_upgrade#configuring-registration-for-fast-forward-upgrades
I've confirmed that the removal of the environment-rhel-registration.yaml and rhel-registration-resource-registry.yaml files from the CLI resulted in a working ffwd upgrade.