Bug 1296335

Summary: Not able to add newly deployed 5.5 worker to the regional DB appliance
Product: Red Hat CloudForms Management Engine Reporter: Jared Deubel <jdeubel>
Component: ApplianceAssignee: Keenan Brock <kbrock>
Status: CLOSED WONTFIX QA Contact: Dave Johnson <dajohnso>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.5.0CC: abellott, fdewaley, jdeubel, jhardy, kbrock, mhild, obarenbo, thenness
Target Milestone: GA   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-21 19:07:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jared Deubel 2016-01-06 22:27:40 UTC 
Description of problem:

Inability to connect to the DB worker from a fresh 5.5 deployment. We see a timeout error (as shown in the snippet below):

On the EVM worker:
===================================================================

[----] I, [2015-12-30T11:05:58.902563 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding MiqWidget
[----] I, [2015-12-30T11:06:06.870573 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding VmdbDatabase
/opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec': execution expired (Timeout::Error)
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
	from /opt/rh/cfme-gemset/gems/activesupport-4.2.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/schema_statements.rb:143:in `indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm.rb:8:in `sql_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table/seeding.rb:6:in `seed_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:17:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:11:in `seed_for_database'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:48:in `block in seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `each'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:7:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:63:in `block (2 levels) in seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `each'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `block in seed'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/transactions.rb:220:in `transaction'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock'
	from /var/www/miq/vmdb/lib/evm_database.rb:51:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:35:in `seed_primordial'
	from /var/www/miq/vmdb/app/models/miq_server.rb:205:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:60:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:79:in `start'
	from /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:3:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>'
	from /var/www/miq/vmdb/bin/rails:4:in `require'
	from /var/www/miq/vmdb/bin/rails:4:in `<main>'
[----] I, [2015-12-30T11:11:39.856060 #13777:445994]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
===================================================================

We have noticed that the increasing sequence count (miq_servers) as a side issue since we think that this is related to how sequences are handled in postgresql, but wanted to point that out.

It seems that we get far enough to increment the sequence and then suddenly it gets rolled back but the sequence can not decrement. 
 
Version-Release number of selected component (if applicable):
5.5 (Customer just recently migrated up from 3.2)
Comment 2 Marcel Hild 2016-01-08 20:35:57 UTC 
Can they run 

script/rails runner tools/db_ping.rb
script/rails runner tools/db_ping_remote.rb

on both, 1) the non-vmdb appliance and 2) the vmdb appliance?

Also, what exact version are they upgrading from and to? (3.2.? -> 5.5.?)
Comment 4 Marcel Hild 2016-01-12 16:36:09 UTC 
Seeding runs within a lock of 10 minutes.

https://github.com/ManageIQ/manageiq/blob/master/lib/evm_database.rb#L51

You might ask your customer to edit that file and replace it with e.g. 

MiqDatabase.with_lock(30.minutes) do

We will discuss how to handle slow network connections.
Comment 13 Keenan Brock 2016-01-29 15:34:45 UTC 
The solution for this is around fixing performance of primordial seeds.

Let me know if this is more than a medium priority.

In the mean time, please run appliances in the same zone as the database
Comment 14 Nick Carboni 2016-02-29 13:52:43 UTC 
*** Bug 1296340 has been marked as a duplicate of this bug. ***
Comment 15 Keenan Brock 2016-03-08 20:02:02 UTC 
*** Bug 1294907 has been marked as a duplicate of this bug. ***
Comment 16 Keenan Brock 2016-04-21 19:07:31 UTC 
We are currently not setup to run a database over the wan