Hide Forgot
Description of problem: Inability to connect to the DB worker from a fresh 5.5 deployment. We see a timeout error (as shown in the snippet below): On the EVM worker: =================================================================== [----] I, [2015-12-30T11:05:58.902563 #13701:9d3988] INFO -- : MIQ(EvmDatabase.seed) Seeding MiqWidget [----] I, [2015-12-30T11:06:06.870573 #13701:9d3988] INFO -- : MIQ(EvmDatabase.seed) Seeding VmdbDatabase /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec': execution expired (Timeout::Error) from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log' from /opt/rh/cfme-gemset/gems/activesupport-4.2.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/postgresql/schema_statements.rb:143:in `indexes' from /var/www/miq/vmdb/app/models/vmdb_table_evm.rb:8:in `sql_indexes' from /var/www/miq/vmdb/app/models/vmdb_table/seeding.rb:6:in `seed_indexes' from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:17:in `seed' from /var/www/miq/vmdb/app/models/vmdb_table_evm/seeding.rb:11:in `seed_for_database' from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:48:in `block in seed' from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `each' from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:46:in `seed' from /var/www/miq/vmdb/app/models/vmdb_database/seeding.rb:7:in `seed' from /var/www/miq/vmdb/lib/evm_database.rb:63:in `block (2 levels) in seed' from /var/www/miq/vmdb/lib/evm_database.rb:52:in `each' from /var/www/miq/vmdb/lib/evm_database.rb:52:in `block in seed' from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock' from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch' from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' from /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout' from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction' from /opt/rh/cfme-gemset/gems/activerecord-4.2.5/lib/active_record/transactions.rb:220:in `transaction' from /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock' from /var/www/miq/vmdb/lib/evm_database.rb:51:in `seed' from /var/www/miq/vmdb/lib/evm_database.rb:35:in `seed_primordial' from /var/www/miq/vmdb/app/models/miq_server.rb:205:in `start' from /var/www/miq/vmdb/lib/workers/evm_server.rb:60:in `start' from /var/www/miq/vmdb/lib/workers/evm_server.rb:79:in `start' from /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:3:in `<top (required)>' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!' from /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>' from /var/www/miq/vmdb/bin/rails:4:in `require' from /var/www/miq/vmdb/bin/rails:4:in `<main>' [----] I, [2015-12-30T11:11:39.856060 #13777:445994] INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN] =================================================================== We have noticed that the increasing sequence count (miq_servers) as a side issue since we think that this is related to how sequences are handled in postgresql, but wanted to point that out. It seems that we get far enough to increment the sequence and then suddenly it gets rolled back but the sequence can not decrement. Version-Release number of selected component (if applicable): 5.5 (Customer just recently migrated up from 3.2)
Can they run script/rails runner tools/db_ping.rb script/rails runner tools/db_ping_remote.rb on both, 1) the non-vmdb appliance and 2) the vmdb appliance? Also, what exact version are they upgrading from and to? (3.2.? -> 5.5.?)
Seeding runs within a lock of 10 minutes. https://github.com/ManageIQ/manageiq/blob/master/lib/evm_database.rb#L51 You might ask your customer to edit that file and replace it with e.g. MiqDatabase.with_lock(30.minutes) do We will discuss how to handle slow network connections.
The solution for this is around fixing performance of primordial seeds. Let me know if this is more than a medium priority. In the mean time, please run appliances in the same zone as the database
*** Bug 1296340 has been marked as a duplicate of this bug. ***
*** Bug 1294907 has been marked as a duplicate of this bug. ***
We are currently not setup to run a database over the wan