Bug 1036549

Description of problem:
Failure is seen when upgrade jenkins-1.4 and jbeapapp-6.0 app or app with jenkins-client-1.4 enabled.

Guess it is relevant with cartridge rename stuff. 
 

Version-Release number of selected component (if applicable):
openshift-enterprise-release-2.0.0c-1.git.2.acc7a39.el6op.noarch.rpm
openshift-enterprise-upgrade-broker-2.0.0c-1.git.2.acc7a39.el6op.noarch.rpm
openshift-enterprise-upgrade-node-2.0.0c-1.git.2.acc7a39.el6op.noarch.rpm
openshift-enterprise-yum-validator-2.0.0c-1.git.2.acc7a39.el6op.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1. Setup ose-1.2 env
2. Create jbeapapp-6.0 app on the node
3. Create app with jenkins-client-1.4 enabled.
4. Upgrade env from 1.2 to 2.0



Actual results:
Failure is seen for these apps.

Seen from /tmp/oo-upgrade/upgrade_errors_node.my-ose.com.cn, get the following error:
{"login":"jialiu","app_name":"jenkins","gear_uuid":"5295cf42aeb90571cc000002","version":"2","errors":["Gear upgrade result is marked incomplete","Failed upgrade after 2 tries"],"warnings":[],"times":{"time_upgrade_on_node_measured_from_broker":93,"time_total_upgrade_gear_measured_from_broker":1385962534770},"hostname":"node.my-ose.com.cn","remote_upgrade_result":{"gear_uuid":"5295cf42aeb90571cc000002","hostname":"node","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{}},"compute_itinerary":{"status":"incomplete","errors":["Unhandled exception performing step: key not found: (jenkins, 1.4, _)\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"],"context":{}}},"upgrade_complete":false,"errors":[{"message":"key not found: (jenkins, 1.4, _)","backtrace":"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"}],"warnings":[],"itinerary":{},"times":{"start_time":1385962534247,"upgrade_on_node_measured_from_node":3},"log":["Beginning 2 upgrade for 5295cf42aeb90571cc000002","Inspecting gear at /var/lib/openshift/5295cf42aeb90571cc000002","Caught an exception during upgrade: key not found: (jenkins, 1.4, _)","Total upgrade time on node (ms): 3"]}}


{"login":"jialiu","app_name":"perlscal","gear_uuid":"5295db35aeb90571cc000175","version":"2","errors":["Gear upgrade result is marked incomplete","Failed upgrade after 2 tries"],"warnings":[],"times":{"time_upgrade_on_node_measured_from_broker":128,"time_total_upgrade_gear_measured_from_broker":1385962951196},"hostname":"node.my-ose.com.cn","remote_upgrade_result":{"gear_uuid":"5295db35aeb90571cc000175","hostname":"node","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{}},"compute_itinerary":{"status":"incomplete","errors":["Unhandled exception performing step: key not found: (jenkins-client, 1.4, _)\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"],"context":{}}},"upgrade_complete":false,"errors":[{"message":"key not found: (jenkins-client, 1.4, _)","backtrace":"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"}],"warnings":[],"itinerary":{},"times":{"start_time":1385962950706,"upgrade_on_node_measured_from_node":4},"log":["Beginning 2 upgrade for 5295db35aeb90571cc000175","Inspecting gear at /var/lib/openshift/5295db35aeb90571cc000175","Creating itinerary entry for incompatible upgrade of redhat:haproxy:1.4:0.0.1","Caught an exception during upgrade: key not found: (jenkins-client, 1.4, _)","Total upgrade time on node (ms): 4"]}}


{"login":"jialiu","app_name":"jbeapapp","gear_uuid":"5295d7e9aeb90571cc000106","version":"2","errors":["Gear upgrade result is marked incomplete","Failed upgrade after 2 tries"],"warnings":[],"times":{"time_upgrade_on_node_measured_from_broker":113,"time_total_upgrade_gear_measured_from_broker":1385962904523},"hostname":"node.my-ose.com.cn","remote_upgrade_result":{"gear_uuid":"5295d7e9aeb90571cc000106","hostname":"node","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{}},"compute_itinerary":{"status":"incomplete","errors":["Unhandled exception performing step: key not found: (jbosseap, 6.0, _)\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"],"context":{}}},"upgrade_complete":false,"errors":[{"message":"key not found: (jbosseap, 6.0, _)","backtrace":"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/cartridge_repository.rb:155:in `select'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:301:in `block (2 levels) in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `block in process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `each'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:1065:in `process_cartridges'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:284:in `block in compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/utils/upgrade_progress.rb:32:in `step'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:278:in `compute_itinerary'\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.0/lib/openshift-origin-node/model/upgrade.rb:154:in `execute'\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:240:in `upgrade_action'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"}],"warnings":[],"itinerary":{},"times":{"start_time":1385962904012,"upgrade_on_node_measured_from_node":7},"log":["Beginning 2 upgrade for 5295d7e9aeb90571cc000106","Inspecting gear at /var/lib/openshift/5295d7e9aeb90571cc000106","Creating itinerary entry for incompatible upgrade of redhat:postgresql:8.4:0.2.0","Caught an exception during upgrade: key not found: (jbosseap, 6.0, _)","Total upgrade time on node (ms): 7"]}}


Expected results:
python and ruby app could be started successfully.

Additional info: