Description of problem: Upgrade ocp(container installed on rhel) with docker-excluder and openshift-excluder installed and enabled in all hosts of the cluster. Upgrade successfully, but docker-excluder is not upgraded. And another issue is openshift-excluder is upgraded on the storage host. before upgrade: openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch after upgrade: openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.5.0.49-1.git.0.c8e072a.el7.noarch openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch Version-Release number of selected component (if applicable): atomic-openshift-utils-3.5.28-1.git.0.103513e.el7.noarch How reproducible: always Steps to Reproduce: 1. Container install ocp on rhel(1 master/node/ectd host + 1 nfs host) 2. Install docker-excluder and openshift-excluder on all hosts in the cluster 3. Prepare ose35 and rhel extra repo on all hosts 4. Upgrade ocp Actual results: Docker-excluder is not upgraded to latest version. Openshift-excluder is upgraded only on storage host. Expected results: Docker-excluder should be upgraded together. Openshift-excluder should not upgraded for containerized env. Additional info:
Upstream PR: https://github.com/openshift/openshift-ansible/pull/3607
Version: atomic-openshift-utils-3.5.32-1.git.0.42cf266.el7.noarch Steps: 1. Container install ocp on rhel(1 master/node/ectd host + 1 nfs host) 2. Install docker-excluder and openshift-excluder on all hosts in the cluster 3. Prepare ose35 and rhel extra repo on all hosts 4. Upgrade ocp Result: Openshift-excluder has been upgraded in all hosts, but docker-excluder still has not been upgraded. before upgrade: openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch after upgrade: openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.5.0.51-1.git.0.d07622a.el7.noarch openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-excluder-3.5.0.51-1.git.0.d07622a.el7.noarch atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
containerized OCP 3.4 cluster with the master HEAD: ==After installation== 10.8.172.108 atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch 10.8.172.110: atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch ==After upgrade== 10.8.172.108: atomic-openshift-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch atomic-openshift-docker-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch 10.8.172.110: atomic-openshift-docker-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch atomic-openshift-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch
Liujia, can you upload the entire log of ansible tasks? To see which tasks are skipped at least. Thanks.
Can you also share the exact sequence of steps you trigger the upgrade? From installing rpms, running openshift-ansible (or ansible-playbook) to sharing inventory files up to uploading the output for the command?
1. Trigger container install of ocp3.4 atomic-openshift-utils-3.4.67-1.git.0.14a0b4d.el7.noarch # ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/config.yml inventory file in comment1 2. Install both of openshift-excluder and docker-excluder # ansible -i hosts all -m shell -a "yum -y install atomic-openshift-excluder" # ansible -i hosts all -m shell -a "yum -y install atomic-openshift-docker-excluder" # ansible -i hosts all -m shell -a "rpm -qa | grep excluder" openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch # ansible -i hosts all -m shell -a "cat /etc/yum.conf|grep exclude" openshift-138.x.x.x | SUCCESS | rc=0 >> exclude= docker*1.20* docker*1.19* docker*1.18* docker*1.17* docker*1.16* docker*1.15* docker*1.14* docker*1.13* tuned-profiles-atomic-openshift-node atomic-openshift-tests atomic-openshift-sdn-ovs atomic-openshift-recycle atomic-openshift-pod atomic-openshift-node atomic-openshift-master atomic-openshift-dockerregistry atomic-openshift-clients-redistributable atomic-openshift-clients atomic-openshift openshift-109.x.x.x | SUCCESS | rc=0 >> exclude= docker*1.20* docker*1.19* docker*1.18* docker*1.17* docker*1.16* docker*1.15* docker*1.14* docker*1.13* tuned-profiles-atomic-openshift-node atomic-openshift-tests atomic-openshift-sdn-ovs atomic-openshift-recycle atomic-openshift-pod atomic-openshift-node atomic-openshift-master atomic-openshift-dockerregistry atomic-openshift-clients-redistributable atomic-openshift-clients atomic-openshift 3. Prepare 3.5 repos on all hosts and change inventory file for upgrade. openshift_release=v3.5 4. Run upgrade palybook atomic-openshift-utils-3.5.32-1.git.0.42cf266.el7.noarch # ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade.yml -vvv|tee upgrade.log 5. after upgrade # ansible -i hosts all -m shell -a "rpm -qa | grep excluder" openshift-138.x.x.x | SUCCESS | rc=0 >> atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch atomic-openshift-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch openshift-109.x.x.x | SUCCESS | rc=0 >> atomic-openshift-excluder-3.5.0.52-1.git.0.844b58f.el7.noarch atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
@Jan Hope above info is helpful. BTW, Could u have a look at my last two comments about excluder card in trello, Thx!
The logs don't say anything about the repos. Can you update roles/openshift_excluder/tasks/disable.yml with the following patch?: diff --git a/roles/openshift_excluder/tasks/disable.yml b/roles/openshift_excluder/tasks/disable.yml index a8deb3e..6a7515a 100644 --- a/roles/openshift_excluder/tasks/disable.yml +++ b/roles/openshift_excluder/tasks/disable.yml @@ -4,6 +4,17 @@ # - with_install # - excluder_package_state # - docker_excluder_package_state +- name: Get available excluder version + command: > + {{ repoquery_cmd }} --qf '%{version}' "{{ openshift.common.service_type }}-docker-excluder" + register: excluder_version + failed_when: false + changed_when: false + +- name: Docker excluder version detected + debug: + msg: "{{ openshift.common.service_type }}-docker-excluder: {{ excluder_version.stdout }}" + - include: init.yml # Install any excluder that is enabled These two tasks will check the docker excluder version that is available before the excluder role tries to update the docker excluder to the latest. Once run, can you then upload the ansible log again? Thanks.
Version: atomic-openshift-utils-3.5.39-1.git.0.cb12cdb.el7.noarch Steps: 1. Container install ocp on rhel(1 master/node/ectd host + 1 nfs host) 2. Install docker-excluder and openshift-excluder on all hosts in the cluster 3. Prepare ose35 and rhel extra repo on all hosts 4. Upgrade ocp Result: All excluders on the node are upgraded to latest version.