| Summary: | 2.0 upgrade mongo migrations fail with replica set | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Luke Meyer <lmeyer> |
| Component: | Cluster Version Operator | Assignee: | Luke Meyer <lmeyer> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 2.0.0 | CC: | bleanhar, jialiu, libra-onpremise-devel |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-02-04 14:52:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
The broker was only configured for one mongo host, so the DataStore class was using MongoClient and connecting to the configured host (which was a slave). https://github.com/openshift/openshift-extras/pull/257 https://github.com/openshift/openshift-extras/commit/2e06aea6c15d3f58af1991d2ddc8061f1d964848 ... and similar for 1.2. openshift.sh was not configuring the broker correctly when the mongodb replicants were installed locally. I would have noticed this broke the brokers too if I had tried a different broker. Not an upgrade bug at all. # wget https://raw.github.com/openshift/openshift-extras/enterprise-1.2/enterprise/install-scripts/generic/openshift.sh Modify openshift.sh: CONF_DATASTORE_REPLICANTS="datastore01.example.com:27017,datastore02.example.com:27017,datastore03.example.com:27017" According to the role of every vm, modify CONF_INSTALL_COMPONENTS. After installation, check broker.conf: MONGO_HOST_PORT="datastore01.example.com:27017,datastore02.example.com:27017,datastore03.example.com:27017" Do upgrade process, it is working well. # ose-upgrade pre INFO: OpenShift broker installed. INFO: Setting broker step 'pre' status to UPGRADING INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre OUTPUT-- INFO: Running upgrade scripts in /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files ran without error: --BEGIN OUTPUT-- + mkdir -p /etc/openshift/upgrade/2/ + for file in /etc/mcollective/client.cfg + command cp -f /etc/mcollective/client.cfg /etc/openshift/upgrade/2/ + cp -f /etc/mcollective/client.cfg /etc/openshift/upgrade/2/ --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops ran without error: --BEGIN OUTPUT-- + oo-admin-clear-pending-ops 0 applications were cleaned up. 0 users were cleaned up. 0 domains were cleaned up. --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease ran without error: --BEGIN OUTPUT-- + /usr/libexec/openshift/ose-upgrade-migrate-datastore --prerelease Starting prerelease migrations. Running migration Migrate_2_0_32... Loading user names into memory Domain members updated: 1 Loading domain ownership into memory Application members updated: 1 Migration Migrate_2_0_32 finished. Running migration Migrate_2_0_36... Migration Migrate_2_0_36 finished. Migrations complete. Done! --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done OUTPUT-- INFO: Setting broker step 'pre' status to COMPLETE INFO: Next step is 'outage' |
Description of problem: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease fails with a replica set. Version-Release number of selected component (if applicable): Steps to Reproduce: 1. Install 1.2 HA infrastructure 2. Run ose-upgrade begin pre Results: ERROR: run_script /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease had errors: --BEGIN OUTPUT-- + /usr/libexec/openshift/ose-upgrade-migrate-datastore --prerelease /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/networking.rb:84:in `send_message_with_gle': 10058: not master (Mongo::ConnectionFailure) from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1100:in `block in insert_documents' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:33:in `block in instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:65:in `instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:32:in `instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1098:in `insert_documents' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1050:in `generate_indexes' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:540:in `create_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:567:in `block in ensure_members_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:566:in `each' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:566:in `ensure_members_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:408:in `migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:11:in `block in migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:9:in `each' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:9:in `migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:1452:in `<main>' Starting prerelease migrations. Running migration Migrate_2_0_32...