Description of problem: ======================= Upgrading from 3.10 latest to 3.11.146 some packages are updated to the latest version 3.11.153 when the value of the openshift_pkg_version is 3.11.146. In the ansible inventory file, the next variables are defined: openshift_release=v3.11 openshift_pkg_version=-3.11.146 openshift_image_tag=v3.11.146 Reviewing the yum history, you can see the next information: [root@master-0 ~]# yum history info 57 Loaded plugins: product-id, search-disabled-repos, subscription-manager Transaction ID : 57 Begin time : Thu Nov 7 05:01:16 2019 Begin rpmdb : 540:e5ff84dcab1f92aba8ca3f488a33ed67e720e33b End time : 05:02:12 2019 (56 seconds) End rpmdb : 540:81f9c2d0ef67dfd3f75dc7e046a4ce5609f3cf54 User : quicklab <quicklab> Return-Code : Success Command Line : -d 2 -y install atomic-openshift-clients-3.11.146 Transaction performed with: Installed rpm-4.11.3-40.el7.x86_64 installed Installed subscription-manager-1.24.13-3.el7_7.x86_64 @rhel-7-server-rpms Installed yum-3.4.3-163.el7.noarch installed Installed yum-metadata-parser-1.1.4-10.el7.x86_64 installed Packages Altered: Updated atomic-openshift-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Updated atomic-openshift-clients-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Updated atomic-openshift-hyperkube-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Updated atomic-openshift-node-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Scriptlet output: 1 warning: /etc/sysconfig/atomic-openshift-node created as /etc/sysconfig/atomic-openshift-node.rpmnew Version-Release number of the following components: =================================================== $ oc version oc v3.11.146 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO $ rpm -q openshift-ansible openshift-ansible-3.11.153-2.git.0.ee699b5.el7.noarch $ rpm -q ansible ansible-2.6.20-1.el7ae.noarch $ ansible --version ansible 2.6.20 config file = /usr/share/ansible/openshift-ansible/ansible.cfg configured module search path = [u'/home/quicklab/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] How reproducible: ================= Always Steps to Reproduce: 1. Upgrade from 3.10 latest to 3.11.146 or another version not being it the latest 2. Check the rpm packages version 3. rpm -qa|grep openshift Actual results: --------------- A mix of rpm OpenShift packages where some of them are from the same version that you have defined in the variable openshift_pkg_version and some of them not. Expected results: ----------------- All rpm packages honour the value of the openshift_pkg_version defined. Additional info: ================ The BZ1391325 [0] and BZ1721406 [1] looks like the same issue [0] https://bugzilla.redhat.com/show_bug.cgi?id=1391325 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1721406
Please attach verbose Ansible logs for the upgrade. A similar issue was addressed in [1] however that change should be present in openshift-ansible-3.11.153-2. The logs will help identify the root cause. [1] https://github.com/openshift/openshift-ansible/pull/11956
Was looking at this today Russell. I think openshift_pkg_version is being applied correctly here. The roles in question are: ./roles/openshift_node/tasks/upgrade/rpm_upgrade.yml ./roles/openshift_node/tasks/upgrade/rpm_upgrade_install.yml openshift_node_upgrade_rpm_list: - "{{ openshift_service_type }}{{ openshift_pkg_version | default('') }}" - "{{ openshift_service_type }}-node{{ openshift_pkg_version | default('') }}" - "{{ openshift_service_type + '-sdn-ovs' + openshift_pkg_version | default('') if openshift_use_openshift_sdn | default(true) | bool else '' }}" - "{{ openshift_service_type }}-clients{{ openshift_pkg_version | default('') }}" - "PyYAML" Looking at the yum history above, it seems like atomic-openshift-clients with the correct package version is causing dependant packages to be updated to the latest version. What is not clear is why / why atomic-openshift-clients would get updated first in it's own transaction, Looks like it's an issue with updating atomic-openshift-clients independently versus an ansible issue with pkg versions. I asked for some more info on yum history from the CU env. Easy test from an old OCP cluster: yum install atomic-openshift-clients-3.11.146* This leads to atomic-openshift being updated to latest and subsequently atomic-openshift-clients being bumped again due to the dep. Loaded plugins: product-id, search-disabled-repos, subscription-manager rhel-7-server-ansible-2.6-rpms | 4.0 kB 00:00:00 rhel-7-server-extras-rpms | 3.4 kB 00:00:00 rhel-7-server-ose-3.11-rpms | 4.1 kB 00:00:00 rhel-7-server-rpms | 3.5 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated --> Processing Dependency: atomic-openshift-clients = 3.11.88-1.git.0.47f4e98.el7 for package: atomic-openshift-3.11.88-1.git.0.47f4e98.el7.x86_64 ---> Package atomic-openshift-clients.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift.x86_64 0:3.11.153-1.git.0.aaf3f71.el7 will be an update --> Processing Dependency: atomic-openshift-clients = 3.11.153-1.git.0.aaf3f71.el7 for package: atomic-openshift-3.11.153-1.git.0.aaf3f71.el7.x86_64 rhel-7-server-ose-3.11-rpms/x86_64/filelists_db | 1.4 MB 00:00:00 --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.11.153-1.git.0.aaf3f71.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================== Updating for dependencies: atomic-openshift x86_64 3.11.153-1.git.0.aaf3f71.el7 rhel-7-server-ose-3.11-rpms 37 M atomic-openshift-clients x86_64 3.11.153-1.git.0.aaf3f71.el7 rhel-7-server-ose-3.11-rpms 23 M Transaction Summary =========================================================================================================================================================================================================== Upgrade ( 2 Dependent packages) If they're all updated together, it *seems* ok... yum install atomic-openshift-3.11.146* atomic-openshift-hyperkube-3.11.146* atomic-openshift-node-3.11.146* atomic-openshift-clients-3.11.146* Loaded plugins: product-id, search-disabled-repos, subscription-manager Resolving Dependencies --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update ---> Package atomic-openshift-hyperkube.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift-hyperkube.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update ---> Package atomic-openshift-node.x86_64 0:3.11.88-1.git.0.47f4e98.el7 will be updated ---> Package atomic-openshift-node.x86_64 0:3.11.146-1.git.0.4aab273.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================== Updating: atomic-openshift x86_64 3.11.146-1.git.0.4aab273.el7 rhel-7-server-ose-3.11-rpms 37 M atomic-openshift-clients x86_64 3.11.146-1.git.0.4aab273.el7 rhel-7-server-ose-3.11-rpms 23 M atomic-openshift-hyperkube x86_64 3.11.146-1.git.0.4aab273.el7 rhel-7-server-ose-3.11-rpms 35 M atomic-openshift-node x86_64 3.11.146-1.git.0.4aab273.el7 rhel-7-server-ose-3.11-rpms 7.2 M Transaction Summary =========================================================================================================================================================================================================== Upgrade 4 Package
Hello Russell, As Matthew is saying, it looks like that ansible is doing everything in the right way. If you read the information provided by the "yum history info 57", then you can see that ansible is asking to update the atomic-openshift-clients to the openshift_pkg_version version that was defined in the inventory file: 3.11.146 ~~~~ [root@master-0 ~]# yum history info 57 Loaded plugins: product-id, search-disabled-repos, subscription-manager Transaction ID : 57 Begin time : Thu Nov 7 05:01:16 2019 Begin rpmdb : 540:e5ff84dcab1f92aba8ca3f488a33ed67e720e33b End time : 05:02:12 2019 (56 seconds) End rpmdb : 540:81f9c2d0ef67dfd3f75dc7e046a4ce5609f3cf54 User : quicklab <quicklab> Return-Code : Success Command Line : -d 2 -y install atomic-openshift-clients-3.11.146 <---------------------------------- this is the same version that was specified in the inventory file Transaction performed with: Installed rpm-4.11.3-40.el7.x86_64 installed Installed subscription-manager-1.24.13-3.el7_7.x86_64 @rhel-7-server-rpms Installed yum-3.4.3-163.el7.noarch installed Installed yum-metadata-parser-1.1.4-10.el7.x86_64 installed Packages Altered: Updated atomic-openshift-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms <---------------- due dependencies, the latest version of atomic-openshift-{clients,hyperkube,node} is installed Updated atomic-openshift-clients-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Updated atomic-openshift-hyperkube-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Updated atomic-openshift-node-3.10.181-1.git.0.3ab4b3d.el7.x86_64 @rhel-7-server-ose-3.10-rpms Update 3.11.153-1.git.0.aaf3f71.el7.x86_64 @rhel-7-server-ose-3.11-rpms Scriptlet output: 1 warning: /etc/sysconfig/atomic-openshift-node created as /etc/sysconfig/atomic-openshift-node.rpmnew ~~~~ Then, I believe that it's not necessary to upload the ansible scripts. If you have an old OCP rpm version installed that are not the latest and you execute: $ yum -d 2 -y install atomic-openshift-clients-3.11.146 You can get the same result that I have obtained and some rpm packages are going to be updated to the latest version. Please, let me now if I can provide you with more information that you need to continue with the Bugzilla
The situation you are describing is the exact problem fixed in [1]. The problem was that the -clients package was being updated in it's own task earlier in the upgrade process[2,3], which was causing the cyclical dependent package upgrade to latest versions. That step was removed and the -clients package was only being updated during the node upgrade where package versions are specified for all dependent packages. Please confirm the version of openshift-ansible installed. The original bug[4] was just verified by QE last night on openshift-ansible-3.11.154-2, which has not yet shipped. Please also attach verbose (-vvv) Ansible logs. The logs will show the output of all the yum transactions which will identify at what point packages are being updated and to which versions and why. [1] https://github.com/openshift/openshift-ansible/pull/11956 [2] https://github.com/openshift/openshift-ansible/pull/11956/files#diff-4d2b31a1102382743aaaa314a787fc5eL164 [3] https://github.com/openshift/openshift-ansible/pull/11956/files#diff-433591899fc94e3ef66b3266d8c24effL2-L8 [4] https://bugzilla.redhat.com/show_bug.cgi?id=1758744
Hello Russel, I have deleted my own environment. I'm recreating a new one to reproduce it again and after that, I'll update the ticket with the ansible logs. As you said, it seems the same that BZ1758744, but we can verified it. Thanks, Oscar
Hello Russel, Sorry, but I have out of office the last days. I have installed the rpm openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch related to the BZ1758744 and I wasn't able to reproduce this issue. Then, it's fixed $ rpm -q openshift-ansible openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch $ rpm -q ansible ansible-2.6.20-1.el7ae.noarch $ ansible --version ansible 2.6.20 config file = /home/quicklab/ansible.cfg configured module search path = [u'/home/quicklab/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] Upgrade to the 3.11.146, all rpm packages are in the right version. 1. Inventory file ~~~ [OSEv3:children] masters nodes etcd lb nfs glusterfs [OSEv3:vars] ansible_ssh_user=quicklab ansible_become=yes debug_level=2 openshift_deployment_type=openshift-enterprise openshift_release=v3.11 openshift_pkg_version=-3.11.146 openshift_image_tag=v3.11.146 oreg_auth_user='username' oreg_auth_password='password' openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider' }] openshift_master_htpasswd_file=~/htpasswd openshift_master_cluster_method=native openshift_master_cluster_hostname=openshift.internal.ocasalsa311.lab.pnq2.cee.redhat.com openshift_master_cluster_public_hostname=openshift.ocasalsa311.lab.pnq2.cee.redhat.com openshift_master_default_subdomain=apps.ocasalsa311.lab.pnq2.cee.redhat.com openshift_master_api_port=443 openshift_master_console_port=443 openshift_override_hostname_check=true openshift_disable_check=disk_availability,docker_storage,package_version [masters] 10.74.177.189 [etcd] 10.74.177.189 [nfs] [lb] 10.74.176.10 [nodes] 10.74.177.189 openshift_node_group_name="node-config-master" 10.74.176.53 openshift_node_group_name="node-config-infra" 10.74.179.85 openshift_node_group_name="node-config-compute" [glusterfs] ~~~ 2. RPM packages after upgrading ~~~ $ ansible all -i hosts -m shell -a "rpm -qa|grep openshift |sort" [WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message. 10.74.179.85 | SUCCESS | rc=0 >> atomic-openshift-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-clients-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-docker-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-hyperkube-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-node-3.11.146-1.git.0.4aab273.el7.x86_64 10.74.176.10 | SUCCESS | rc=0 >> 10.74.176.53 | SUCCESS | rc=0 >> atomic-openshift-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-clients-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-docker-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-hyperkube-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-node-3.11.146-1.git.0.4aab273.el7.x86_64 10.74.177.189 | SUCCESS | rc=0 >> atomic-openshift-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-clients-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-docker-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-excluder-3.11.146-1.git.0.4aab273.el7.noarch atomic-openshift-hyperkube-3.11.146-1.git.0.4aab273.el7.x86_64 atomic-openshift-node-3.11.146-1.git.0.4aab273.el7.x86_64 openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch openshift-ansible-docs-3.11.154-2.git.0.1640c49.el7.noarch openshift-ansible-playbooks-3.11.154-2.git.0.1640c49.el7.noarch openshift-ansible-roles-3.11.154-2.git.0.1640c49.el7.noarch ~~~
Oscar, Thanks for your verification. *** This bug has been marked as a duplicate of bug 1758744 ***