Description of problem: Noticed that our environments were not withstanding shutdown of the first mongo server, I used the rails console to see the mongo hosts and see only the first is being configured. Here is the /etc/openshift/broker.conf [root@Abroker1 openshift]# cat broker.conf # Domain suffix to use for applications (Must match node config) CLOUD_DOMAIN="osop.cloud.lab.eng.bos.redhat.com" # Comma seperted list of valid gear sizes VALID_GEAR_SIZES="small,medium" # Default number of gears to assign to a new user DEFAULT_MAX_GEARS="100" # Default gear size for a new gear DEFAULT_GEAR_SIZE="small" #Broker datastore configuration MONGO_REPLICA_SETS=true # Replica set example: "<host-1>:<port-1> <host-2>:<port-2> ..." MONGO_HOST_PORT="Absn1.osop.cloud.lab.eng.bos.redhat.com:27017 Absn2.osop.cloud.lab.eng.bos.redhat.com:27017 Absn3.osop.cloud.lab.eng.bos.redhat.com:27017" MONGO_USER="openshift" MONGO_PASSWORD="mooo" MONGO_DB="openshift_broker_dev" #Enables gear/filesystem resource usage tracking ENABLE_USAGE_TRACKING_DATASTORE="false" #Log resource usage information to syslog ENABLE_USAGE_TRACKING_SYSLOG="false" #Enable all broker analytics ENABLE_ANALYTICS="false" #Enables logging of REST API operations and success/failure ENABLE_USER_ACTION_LOG="true" USER_ACTION_LOG_FILE="/var/log/openshift/user_action.log" AUTH_SALT="ClWqe5zKtEW4CJEMyjzQ" AUTH_PRIVKEYFILE="/etc/openshift/server_priv.pem" AUTH_PRIVKEYPASS="" AUTH_PUBKEYFILE="/etc/openshift/server_pub.pem" AUTH_RSYNC_KEY_FILE="/etc/openshift/rsync_id_rsa" [root@Abroker1 openshift]# Here is the values I saw from a rail console: cd /var/www/openshift/broker [root@Abroker1 broker]# rails console Loading production environment (Rails 3.0.13) irb(main):001:0> store = OpenShift::DataStore.instance => #<OpenShift::MongoDataStore:0x7f3fe4156460 @host_port=["Absn1.osop.cloud.lab.eng.bos.redhat.com", 27017], @db="openshift_broker_dev", @user="openshift", @replica_set=true, @password="mooo", @collections={:application_template=>"template", :user=>"user", :district=>"district"}> irb(main):002:0> exit [root@Abroker1 broker]# Brenton had me check the value of replica_sets: [root@Abroker1 broker]# su -l apache --shell=/bin/bash [apache@Abroker1 ~]$ cd /var/www/openshift/broker/ [apache@Abroker1 broker]$ ./script/rails console Loading production environment (Rails 3.0.13) irb(main):001:0> Rails.configuration.auth[:mongo_replica_sets] => nil irb(main):002:0> Version-Release number of selected component (if applicable): [apache@Abroker1 broker]$ uname -a ; cat /etc/redhat-release Linux Abroker1.osop.cloud.lab.eng.bos.redhat.com 2.6.32-279.14.1.el6.x86_64 #1 SMP Mon Oct 15 13:44:51 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.3 (Santiago) [apache@Abroker1 broker]$ yum list installed | grep -i -A 1 openshift *Note* Red Hat Network repositories are not listed below. You must run this command as root to access RHN repositories. openshift-console.noarch 0.0.5-3.el6 @rhel-x86_64-server-6-osop-1-infrastructure openshift-origin-broker.noarch 1.0.2-1.el6 @rhel-x86_64-server-6-osop-1-infrastructure openshift-origin-broker-util.noarch 1.0.5-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure openssh.x86_64 5.3p1-81.el6_3 @rhel-x86_64-server-6 -- rubygem-openshift-origin-auth-remote-user.noarch 1.0.3-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure rubygem-openshift-origin-common.noarch 1.0.1-2.el6op @rhel-x86_64-server-6-osop-1-infrastructure rubygem-openshift-origin-console.noarch 1.0.3-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure rubygem-openshift-origin-controller.noarch 1.0.7-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure rubygem-openshift-origin-dns-bind.noarch 1.0.1-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure rubygem-openshift-origin-msg-broker-mcollective.noarch 1.0.2-1.el6op @rhel-x86_64-server-6-osop-1-infrastructure [apache@Abroker1 broker]$ How reproducible: easily in 2 separate configurations Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Problem is the "elif" at line 115 of production.rb - if !hp raise "Broker is missing Mongo configuration." elif replica_sets "elif" isn't a keyword, so it's considered a method and the rest of that section is dead code. The logic goes to the "else" clause every time. Guess that's why we have coverage tests :) Change it to "elsif" to fix. BTW oo-accept-broker doesn't handle this properly.
Did I say 115? I meant line 45. I'm going to bed now.
Ha, great catch Luke. This is fixed in the next puddle. http://download.devel.redhat.com/brewroot/packages/openshift-origin-broker/1.0.6/1.el6op/noarch/openshift-origin-broker-1.0.6-1.el6op.noarch.rpm I'll make sure this is fixed upstream too.
[root@broker broker]# cat /etc/openshift/broker.conf | grep MONGO_REPLICA_SETS MONGO_REPLICA_SETS=true [root@broker broker]# cat /etc/openshift/broker.conf | grep MONGO_HOST_PORT #MONGO_HOST_PORT="localhost:27017" MONGO_HOST_PORT="Absn1.osop.cloud.lab.eng.bos.redhat.com:27017 Absn2.osop.cloud.lab.eng.bos.redhat.com:27017 Absn3.osop.cloud.lab.eng.bos.redhat.com:27017" [root@broker broker]# rails console Loading production environment (Rails 3.0.13) irb(main):001:0> store = OpenShift::DataStore.instance => #<OpenShift::MongoDataStore:0x7f6c55d87610 @replica_set=true, @db="openshift_broker", @user="openshift", @collections={:application_template=>"template", :user=>"user", :district=>"district"}, @password="mongopass", @host_port=[["Absn1.osop.cloud.lab.eng.bos.redhat.com", 27017], ["Absn2.osop.cloud.lab.eng.bos.redhat.com", 27017], ["Absn3.osop.cloud.lab.eng.bos.redhat.com", 27017]]> irb(main):002:0> exit Verify: OpenShiftEnterprise/1.1.x/2012-12-18.1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0220.html