Hide Forgot
Description of problem: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease fails with a replica set. Version-Release number of selected component (if applicable): Steps to Reproduce: 1. Install 1.2 HA infrastructure 2. Run ose-upgrade begin pre Results: ERROR: run_script /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease had errors: --BEGIN OUTPUT-- + /usr/libexec/openshift/ose-upgrade-migrate-datastore --prerelease /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/networking.rb:84:in `send_message_with_gle': 10058: not master (Mongo::ConnectionFailure) from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1100:in `block in insert_documents' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:33:in `block in instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:65:in `instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/util/logging.rb:32:in `instrument' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1098:in `insert_documents' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:1050:in `generate_indexes' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:540:in `create_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:567:in `block in ensure_members_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:566:in `each' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:566:in `ensure_members_index' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:408:in `migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:11:in `block in migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:9:in `each' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:9:in `migrate' from /usr/libexec/openshift/ose-upgrade-migrate-datastore:1452:in `<main>' Starting prerelease migrations. Running migration Migrate_2_0_32...
The broker was only configured for one mongo host, so the DataStore class was using MongoClient and connecting to the configured host (which was a slave).
https://github.com/openshift/openshift-extras/pull/257 https://github.com/openshift/openshift-extras/commit/2e06aea6c15d3f58af1991d2ddc8061f1d964848 ... and similar for 1.2. openshift.sh was not configuring the broker correctly when the mongodb replicants were installed locally. I would have noticed this broke the brokers too if I had tried a different broker. Not an upgrade bug at all.
# wget https://raw.github.com/openshift/openshift-extras/enterprise-1.2/enterprise/install-scripts/generic/openshift.sh Modify openshift.sh: CONF_DATASTORE_REPLICANTS="datastore01.example.com:27017,datastore02.example.com:27017,datastore03.example.com:27017" According to the role of every vm, modify CONF_INSTALL_COMPONENTS. After installation, check broker.conf: MONGO_HOST_PORT="datastore01.example.com:27017,datastore02.example.com:27017,datastore03.example.com:27017" Do upgrade process, it is working well. # ose-upgrade pre INFO: OpenShift broker installed. INFO: Setting broker step 'pre' status to UPGRADING INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre OUTPUT-- INFO: Running upgrade scripts in /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files ran without error: --BEGIN OUTPUT-- + mkdir -p /etc/openshift/upgrade/2/ + for file in /etc/mcollective/client.cfg + command cp -f /etc/mcollective/client.cfg /etc/openshift/upgrade/2/ + cp -f /etc/mcollective/client.cfg /etc/openshift/upgrade/2/ --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-backup-conf-files OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops ran without error: --BEGIN OUTPUT-- + oo-admin-clear-pending-ops 0 applications were cleaned up. 0 users were cleaned up. 0 domains were cleaned up. --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/02-broker-clear-most-pending-ops OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease ran without error: --BEGIN OUTPUT-- + /usr/libexec/openshift/ose-upgrade-migrate-datastore --prerelease Starting prerelease migrations. Running migration Migrate_2_0_32... Loading user names into memory Domain members updated: 1 Loading domain ownership into memory Application members updated: 1 Migration Migrate_2_0_32 finished. Running migration Migrate_2_0_36... Migration Migrate_2_0_36 finished. Migrations complete. Done! --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/04-broker-migrate-datastore-prerelease OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/pre/05-broker-add-authorizations-indexes OUTPUT-- INFO: running /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done INFO: /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done ran without error: --BEGIN OUTPUT-- --END /usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/2/step_lock oseupgrade_2_pre done OUTPUT-- INFO: Setting broker step 'pre' status to COMPLETE INFO: Next step is 'outage'