Description of problem: Cartridge management has become more complicated with OSE 2.1. It is legitimate now to have different sets of cartridges available per profile (although the broker isn't per-profile-aware until OSE 2.2). Moving a gear fails when nodes have different cartridge versions. Steps to Reproduce: 1. Deploy OSE with multiple gear profiles with multiple nodes 2. Make sure all nodes from at least one profile are missing a cartridge present in the others 3. Make sure one node from a profile is missing a cartridge present on another node of the same profile 4. Make sure one node has a cartridge with a different version from other nodes providing the same cartridge name. 5. Ensure cartridges have been imported and activated at the broker for every different set of cartridges. 6. Run oo-accept-systems -v Actual results: Failure reports about #2 and #3, not #4 Expected results: Failure reports about #3 and #4, not #1 (however with -v the difference is noted as INFO level) Additional info: https://github.com/openshift/origin-server/pull/5851 implements the desired check. I would like to have QE check for edge cases before porting to OSE 2.1. Under 2.2 it will also check according to whether the broker.conf:VALID_GEAR_SIZES_FOR_CARTRIDGE. Also note that downloadable cartridges imported and cartridges disabled at the broker are removed from consideration for consistency checks, and a cartridge with a different vendor is also considered a different version of the same-named cartridge.
This fix incorporates the following commits from origin-server: commit 2ff03e477686c7145fefda3d5d0b085a0cb51666 Author: Miciah Masters <miciah.masters> Date: Fri Jul 11 17:40:02 2014 -0400 oo-accept-systems: Better output in carts check Improve the output in the check_nodes_cartridges check of oo-accept-systems: Sort cartridges before printing in order to make it easier to compare lists, and use consistent formatting. commit 9b38b05e6bd76634991479aaa0b162070812b06d Author: Miciah Masters <miciah.masters> Date: Wed Jul 30 12:32:20 2014 -0400 oo-accept-systems: Drop obsolete/downloaded carts Fix the check_nodes_cartridges test in oo-accept-systems to ignore obsolete cartridges returned by the node hosts, because the broker will not list obsolete cartridges, and to ignore downloaded cartridges returned by the by the broker, because a node will not have downloaded cartridges in its cartridge repository. This commit fixes bug 1124938. commit 7b0428d394ecc9e6e72b10fc1aea1af56986b26c Author: Miciah Masters <miciah.masters> Date: Wed Jul 30 12:59:00 2014 -0400 oo-accept-systems: Warn about obsolete cartridges Modify the check_nodes_cartridges test in oo-accept-systems to warn about obsolete cartridges returned by the node hosts. commit 2bfadcf26196a2ea681f743cc27abb81d4de0b8e Author: Luke Meyer <lmeyer> Date: Fri Oct 3 23:24:29 2014 -0400 oo-accept-systems: improve cartridge integrity checks Cartridge management has become more complicated. It is legitimate now to have different sets of cartridges available per profile. Moving gears fails when nodes have different cartridge versions. oo-accept-systems now checks that all nodes in a profile supply a consistent set of cartridges, and that profiles supply cartridges expected by the broker. Cartridge version differences are also detected. Enterprise trello card https://trello.com/c/cGT5PJJt/277-3-xpaas-preparedness commit 207721a1097131bd4deb12256bdd51ef8a920ef7 Author: Luke Meyer <lmeyer> Date: Mon Oct 6 15:38:03 2014 -0400 oo-accept-systems: fix errors from PR 5851 The cartridge consistency checks introduced in PR 5851 did not correctly determine which cartridges were active on the broker, which becomes apparent once you have imported more than one version of any cartridge. Inactive versions of the cartridge could be compared against the node versions. This is now fixed. Also obsoletes were added to the list of things not to compare, and probably some other issues addressed along the way.
check on puddle [2.1.z/2014-11-05.1] install multi-nodes env: broker medium profile node1 small profile node2 small profile 1. erase php python ruby on broker #rpm -e openshift-origin-cartridge-python-1.23.4.7-1.el6op.noarch openshift-origin-cartridge-php-1.23.3.7-1.el6op.noarch openshift-origin-cartridge-ruby-1.23.3.7-1.el6op.noarch #oo-admin-cartridge -a erase --name python --cartridge_version 0.0.17.1 --version 2.6 #oo-admin-cartridge -a erase --name ruby --cartridge_version 0.0.18.1 --version 1.8 #oo-admin-cartridge -a erase --name php --cartridge_version 0.0.16.1 --version 5.4 2. erase msyql postgresql on node1 #rpm -e openshift-origin-cartridge-postgresql-1.23.3.4-1.el6op.noarch openshift-origin-cartridge-mysql-1.23.4.5-1.el6op.noarch #oo-admin-cartridge -a erase --name msyql --cartridge_version 0.2.13.2 --version 5.1 #oo-admin-cartridge -a erase --name postgresql --cartridge_version 0.3.14.1 --version 8.4 3. modify the nodejs version on node2 #vim /usr/libexec/openshift/cartridges/nodejs/metadata/manmanifest.yml - Cartridge-Version: 0.0.17.3 + Cartridge-Version: 0.1.17.3 # oo-admin-cartridge -a install 0.1.17.3/--source /usr/libexec/openshift/cartridges/nodejs/ #oo-admin-cartridge -a erase -n nodejs --cartridge_version 0.0.17.3 --version 0.10 4. import cartridge on broker oo-admin-ctl-cartridge -c import-node --node broker.ose21z-auto.com.cn --activate --obsolete oo-admin-ctl-cartridge -c import-node --node node1.ose21z-auto.com.cn --activate --obsolete oo-admin-ctl-cartridge -c import-node --node node2.ose21z-auto.com.cn --activate --obsolete 5. run 'oo-accept-systems -v' on broker #oo-accept-systems -v Output FAIL: There were mismatches in cartridges available on nodes: node 'node1.ose21z-auto.com.cn' cartridge list is missing mysql-5.1, mysql-5.5, postgresql-8.4, postgresql-9.2 compared to the profile 'small' All nodes in a profile should have the same set of cartridges installed. Restart mcollective after installing new cartridges on a node. To import new cartridges into the broker: # oo-admin-ctl-cartridge -c import-profile --activate FAIL: Some cartridge versions on the broker and nodes did not match. If nodes have different cartridge versions, moving gears between them may fail, and there may be other inconsistencies. Cartridge versions are also used to determine when gears need upgrades and whether restarts are required. Ensure that all nodes have the latest cartridge versions installed. Restart mcollective on each node after updating cartridges. For node 'broker.ose21z-auto.com.cn', the cartridge versions do not match what the broker has active: Broker has: redhat-nodejs-0.10-0.1.17.3 Node has: redhat-nodejs-0.10-0.0.17.3 For node 'node1.ose21z-auto.com.cn', the cartridge versions do not match what the broker has active: Broker has: redhat-nodejs-0.10-0.1.17.3 Node has: redhat-nodejs-0.10-0.0.17.3 To import updated cartridges on the broker and upgrade existing gears: # oo-admin-ctl-cartridge -c import-profile --activate # oo-admin-ctl-cartridge -c migrate # oo-admin-upgrade archive # oo-admin-upgrade upgrade-gear --version <version> 2 ERRORS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2014-1906.html