Bug 1296340

Summary: Not able to add newly deployed 5.5 worker to the regional DB appliance
Product: Red Hat CloudForms Management Engine Reporter: Jared Deubel <jdeubel>
Component: ApplianceAssignee: Gregg Tanzillo <gtanzill>
Status: CLOSED DUPLICATE QA Contact: Alex Newman <anewman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.5.0CC: abellott, jhardy, ncarboni, obarenbo, sshveta
Target Milestone: GA   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-29 13:52:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jared Deubel 2016-01-06 22:41:40 UTC
Description of problem:

Inability to connect to the DB worker from a fresh 5.5 deployment. We see a timeout error (as shown in the snippet below):

On the EVM worker:
===================================================================

[----] I, [2015-12-30T11:05:58.902563 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding MiqWidget
[----] I, [2015-12-30T11:06:06.870573 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding VmdbDatabase
/opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec': execution expired (Timeout::Error)
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
	from /opt/rh/cfme-gemset/gems/activesupport-4.2.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/schema_statements.rb:143:in `indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm.rb:8:in `sql_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table/seeding.rb:6:in `seed_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:17:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:11:in `seed_for_database'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:48:in `block in seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `each'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:7:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:63:in `block (2 levels) in seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `each'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `block in seed'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/transactions.rb:220:in `transaction'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock'
	from /var/www/miq/vmdb/lib/evm_database.rb:51:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:35:in `seed_primordial'
	from /var/www/miq/vmdb/app/models/miq_server.rb:205:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:60:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:79:in `start'
	from /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:3:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>'
	from /var/www/miq/vmdb/bin/rails:4:in `require'
	from /var/www/miq/vmdb/bin/rails:4:in `<main>'
[----] I, [2015-12-30T11:11:39.856060 #13777:445994]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
===================================================================

We have noticed that the increasing sequence count (miq_servers) as a side issue since we think that this is related to how sequences are handled in postgresql, but wanted to point that out.

It seems that we get far enough to increment the sequence and then suddenly it gets rolled back but the sequence can not decrement. 
 
Version-Release number of selected component (if applicable):
5.5 (Customer just recently migrated up from 3.2)

Comment 2 Nick Carboni 2016-02-29 13:52:43 UTC

*** This bug has been marked as a duplicate of bug 1296335 ***