Bug 1191283
| Summary: | Duplicate cartridges are seen when importing active cartridges | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Eric Rich <erich> |
| Component: | Node | Assignee: | Timothy Williams <tiwillia> |
| Status: | CLOSED ERRATA | QA Contact: | libra bugs <libra-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 2.2.0 | CC: | adellape, bleanhar, cryan, jokerman, libra-onpremise-devel, lmeyer, mmccomas, tiwillia, xiama |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | rubygem-openshift-origin-controller-1.37.3.1-1.el6op | Doc Type: | Bug Fix |
| Doc Text: |
When a cartridge was imported into the broker's database from two separate nodes within the same second, the second cartridge failed activation because the time stamp of the first cartridge was seen as the same as the current time. Instead of comparing the time stamps through the moped database query, which only compares seconds, this bug fix updates the logic to compare the Time objects with Ruby. As a result, the priority time stamps are compared to the milliseconds, allowing multiple cartridges from different nodes to be imported and activated within the same second.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-09-30 16:36:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/b1e1d1d1d909328fad7f8803808bad93bc61d4d5 Resolve race condition where cartridges activated within the same second conflicted Bug 1191283 Bugzilla link https://bugzilla.redhat.com/show_bug.cgi?id=1191283 Using the ":priority.lt => self.priority" would not find types with a priority within the same second of this types priority. Because of this, older types did not have their priority properly set to 'nil'. This caused us to believe that a cartridge type with the same name was activated at the same time (or after) this type. Check on rubygem-openshift-origin-controller-1.36.2.3-1.el6op.noarch QE can not reproduce this issue. move it to VERIFIED. Check on puddle [2015-09-17] Update or install a cartridge on all nodes unless the cartridge list has not yet been imported on the broker. [root@broker conf.d]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. Updating 1 cartridges ... 55fba00b82611d04fa000001 # A perl-5.10 (active) Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node3.ose22-auto.com.cn'. No changes [root@broker conf.d]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node3.ose22-auto.com.cn'. No changes Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes [root@broker conf.d]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node3.ose22-auto.com.cn'. No changes Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes [root@broker conf.d]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node3.ose22-auto.com.cn'. No changes Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes There is no duplicate cartridge. Check on puddle [2.2.7/2015-09-22.1] [root@broker ~]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. Updating 1 cartridges ... 5602054282611d93b7000001 # A perl-5.10 (active) Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node1.ose22-auto.com.cn'. No changes [root@broker ~]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node1.ose22-auto.com.cn'. No changes [root@broker ~]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node2.ose22-auto.com.cn'. No changes Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes [root@broker ~]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node1.ose22-auto.com.cn'. No changes [root@broker ~]# oo-admin-ctl-cartridge -c import-profile --activate Importing cartridges for profile 'medium' and platform 'linux'. Importing cartridges from node 'broker.ose22-auto.com.cn'. No changes Importing cartridges for profile 'small' and platform 'linux'. Importing cartridges from node 'node2.ose22-auto.com.cn'. No changes There is no duplicate cartridge. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1844.html |
Description of problem: The OpenShift Console / RHC can see duplicate cartridges when running oo-admin-ctl-cartridge -c import-profile --activate. Version-Release number of selected component (if applicable): 2.1 and 2.2 How reproducible: 50% (guess) Steps to Reproduce: * 1. oo-admin-ctl-cartridge -c import-profile --activate - Run this multiple times. Verification can be done with: 2. oo-admin-console-cache --clear oo-admin-broker-cache --clear 3. oo-admin-ctl-cartridge -c list - This shows duplicate cartridges some with the same TIME (Priority) rhc cartridges - This shows duplicate cartridges Actual results: What ends up happening after running this command a few times is something like the following: This is seen when importing the cartridges. Updating 2 cartridges ... Failed to update ruby-1.9 Another cartridge version 54da53802f6c6ec8be000001 was activated at 2015-02-10T18:52:48+00:00. However when you run (the following) you see a list of cartridges with different ID's # oo-admin-ctl-cartridge --raw -c list | grep -e Display-Name -e Id -e Cartridge-Version "Display-Name": "Ruby 1.9", "Id": {"$oid": "54da53802f6c6ec8be000001"}, "Cartridge-Version": "0.0.18.1", "Display-Name": "Ruby 1.9", "Id": {"$oid": "54da53802f6c6ec8be000002"}, Expected results: You should not import cartridges of the same NAME / Version. Additional info: https://github.com/openshift/enterprise-server/blob/enterprise-2.2/controller/app/models/cartridge_type.rb#L111-L127 The issue seems to be with the TIME (Priority) and how this is determined. Currently the following is the only effective way to correct the duplicates: # for cartridge in $(oo-admin-ctl-cartridge -c list | awk '{print $2}'); do oo-admin-ctl-cartridge -c delete -n $cartridge; done # oo-admin-ctl-cartridge -c import-profile --activate # oo-admin-console-cache --clear # oo-admin-broker-cache --clear