Bug 1296335 - Not able to add newly deployed 5.5 worker to the regional DB appliance
Not able to add newly deployed 5.5 worker to the regional DB appliance
Status: CLOSED WONTFIX
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance (Show other bugs)
5.5.0
Unspecified Unspecified
medium Severity medium
: GA
: 5.6.0
Assigned To: Keenan Brock
Dave Johnson
:
: 1294907 1296340 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-06 17:27 EST by Jared Deubel
Modified: 2016-04-21 15:07 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-21 15:07:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jared Deubel 2016-01-06 17:27:40 EST
Description of problem:

Inability to connect to the DB worker from a fresh 5.5 deployment. We see a timeout error (as shown in the snippet below):

On the EVM worker:
===================================================================

[----] I, [2015-12-30T11:05:58.902563 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding MiqWidget
[----] I, [2015-12-30T11:06:06.870573 #13701:9d3988]  INFO -- : MIQ(EvmDatabase.seed) Seeding VmdbDatabase
/opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec': execution expired (Timeout::Error)
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
	from /opt/rh/cfme-gemset/gems/activesupport-4.2.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/schema_statements.rb:143:in `indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm.rb:8:in `sql_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table/seeding.rb:6:in `seed_indexes'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:17:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:11:in `seed_for_database'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:48:in `block in seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `each'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `seed'
	from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:7:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:63:in `block (2 levels) in seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `each'
	from /var/www/miq/vmdb/lib/evm_database.rb:52:in `block in seed'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
	from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
	from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/transactions.rb:220:in `transaction'
	from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock'
	from /var/www/miq/vmdb/lib/evm_database.rb:51:in `seed'
	from /var/www/miq/vmdb/lib/evm_database.rb:35:in `seed_primordial'
	from /var/www/miq/vmdb/app/models/miq_server.rb:205:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:60:in `start'
	from /var/www/miq/vmdb/lib/workers/evm_server.rb:79:in `start'
	from /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:3:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>'
	from /var/www/miq/vmdb/bin/rails:4:in `require'
	from /var/www/miq/vmdb/bin/rails:4:in `<main>'
[----] I, [2015-12-30T11:11:39.856060 #13777:445994]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
===================================================================

We have noticed that the increasing sequence count (miq_servers) as a side issue since we think that this is related to how sequences are handled in postgresql, but wanted to point that out.

It seems that we get far enough to increment the sequence and then suddenly it gets rolled back but the sequence can not decrement. 
 
Version-Release number of selected component (if applicable):
5.5 (Customer just recently migrated up from 3.2)
Comment 2 Marcel Hild 2016-01-08 15:35:57 EST
Can they run 

script/rails runner tools/db_ping.rb
script/rails runner tools/db_ping_remote.rb

on both, 1) the non-vmdb appliance and 2) the vmdb appliance?

Also, what exact version are they upgrading from and to? (3.2.? -> 5.5.?)
Comment 4 Marcel Hild 2016-01-12 11:36:09 EST
Seeding runs within a lock of 10 minutes.

https://github.com/ManageIQ/manageiq/blob/master/lib/evm_database.rb#L51

You might ask your customer to edit that file and replace it with e.g. 

MiqDatabase.with_lock(30.minutes) do

We will discuss how to handle slow network connections.
Comment 13 Keenan Brock 2016-01-29 10:34:45 EST
The solution for this is around fixing performance of primordial seeds.

Let me know if this is more than a medium priority.

In the mean time, please run appliances in the same zone as the database
Comment 14 Nick Carboni 2016-02-29 08:52:43 EST
*** Bug 1296340 has been marked as a duplicate of this bug. ***
Comment 15 Keenan Brock 2016-03-08 15:02:02 EST
*** Bug 1294907 has been marked as a duplicate of this bug. ***
Comment 16 Keenan Brock 2016-04-21 15:07:31 EDT
We are currently not setup to run a database over the wan

Note You need to log in before you can comment on or make changes to this bug.