Bug 1662126 - CloudTenant is deleted by provider refresh
Summary: CloudTenant is deleted by provider refresh
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.10.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: GA
: 5.10.0
Assignee: Sam Lucidi
QA Contact: Yadnyawalk Tale
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On:
Blocks: 1601090
TreeView+ depends on / blocked
 
Reported: 2018-12-26 10:00 UTC by Fabien Dupont
Modified: 2019-02-12 16:52 UTC (History)
13 users (show)

Fixed In Version: 5.10.0.31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-12 16:52:40 UTC
Category: ---
Cloudforms Team: Openstack
Target Upstream Version:


Attachments (Terms of Use)
Screenshot showing mangled mapping (13.24 KB, image/png)
2018-12-26 10:00 UTC, Fabien Dupont
no flags Details

Description Fabien Dupont 2018-12-26 10:00:08 UTC
Created attachment 1516790 [details]
Screenshot showing mangled mapping

Description of problem:
During VM Migration, the OpenStack provider is refreshed and this deletes the destination CloudTenant from the inventory. This leads to a failure of the migration.

In the Migration UI, I have created a new mapping, named 'OSP Mapping', with 'Corp Website' as destination CloudTenant. Then, I've created a plan using the mapping. When running that plan, it fails with the following error (automation.log):

[----] E, [2018-12-26T03:49:03.432217 #22761:d74f308] ERROR -- : Q-task_id([r10000000000031_service_template_transformation_plan_task_10000000000029]) Method STDERR: (drbunix:///tmp/automation_engine20181226-22761-1lj8znc) /var/www/miq/vmdb/app/models/service_template_transformation_plan_task.rb:70:in `destination_cluster': [fdupont-test-migration] Cluster #<EmsCluster:0x000000001ac7e060> has no mapping. (RuntimeError)

And the error is 'normal', because the source cluster has no destination mapping:

irb> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster'))
=> nil

Looking at the mapping itself, I can see that there's still a reference to the destination tenant:

irb> TransformationMapping.find_by(name: 'OSP Mapping').transformation_mapping_items.select { |tmi| tmi.source_type == 'EmsCluster' }
=> [#<TransformationMappingItem id: 10000000000037, source_id: 10000000000001, source_type: "EmsCluster", destination_id: 10000000000009, destination_type: "CloudTenant", transformation_mapping_id: 10000000000007, options: {}, created_at: "2018-12-26 09:17:59", updated_at: "2018-12-26 09:17:59">]

But the CloudTenant with id 10000000000009 doesn't exist:

irb> CloudTenant.find(10000000000009)
ActiveRecord::RecordNotFound: Couldn't find CloudTenant with 'id'=10000000000009

Looking for the same tenant by name also returns nil:

irb> CloudTenant.find_by(name: 'Corp Website')
=> nil

Running a full refresh of the provider brings the CloudTenant back in the inventory, but with a different id:

irb> CloudTenant.find_by(name: 'Corp Website')
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000017, name: "Corp Website", description: "", enabled: true, ems_ref: "ce5654dbd92d467ea135ca4feb9ccee8", ems_id: 10000000000006, created_at: "2018-12-26 09:46:12", updated_at: "2018-12-26 09:46:12", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

So, it seems that the targeted inventory refresh deletes the original CloudTenant.

In the UI, I can also see that the mapping is wrong: cf. attached screenshot.

Comment 2 Ladislav Smola 2019-01-03 13:34:16 UTC
I couldn't verify that targeted refresh would cause this behavior. Can we repeat the experiment, to find out if it's always reproducible? (I've switched to :debug on the testing appliance)

Comment 3 Fabien Dupont 2019-01-04 08:32:02 UTC
I'm able to reproduce 100%. I've created a mapping and a plan, then run the migration, using the following commands in Rails console:

irb> osp_mapping = TransformationMapping.create(:name => 'BZ1662126 Mapping')
irb> source_vm = Vm.find_by(:name => "fdupont-test-migration", :vendor => 'vmware')
irb> source_cluster = source_vm.ems_cluster
irb> destination_cluster = CloudTenant.find_by(:name => 'HR Intranet')
irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_cluster, :destination => destination_cluster)
irb> source_network = source_vm.hardware.nics.first.lan
irb> destination_network = CloudNetwork.find_by_name('external')
irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_network, :destination => destination_network)
irb> source_storage = source_vm.hardware.disks.select { |d| d.device_type == 'disk' }.first.storage
irb> destination_storage = CloudVolumeType.find_by(name: 'default')
irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_storage, :destination => destination_storage)
irb> destination_flavor = Flavor.find_by(name: 'x1.small')
irb> destination_security_group = SecurityGroup.find_by(name: 'default', cloud_tenant_id: destination_cluster.id)
irb> osp_plan = ServiceTemplateTransformationPlan.create_catalog_item(:name => "BZ1662126 Plan", :config_info => { :transformation_mapping_id => osp_mapping.id, :actions => [ { :vm_id => source_vm.id.to_s, :osp_flavor_id => destination_flavor.id, :osp_security_group_id => destination_security_group.id } ] })
irb> osp_plan.provision_request(User.first)

Before the inventory refresh, I can get the destination_cluster (which is a CloudTenant) from the DB:

irb> destination_cluster.reload
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000016, name: "HR Intranet", description: "", enabled: true, ems_ref: "39f6ca6e7cdd40278d67394d8144c2e7", ems_id: 10000000000006, created_at: "2018-12-26 09:25:55", updated_at: "2018-12-26 09:25:55", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

After the refresh, here is the error I get:

irb> destination_cluster.reload
ActiveRecord::RecordNotFound: Couldn't find ManageIQ::Providers::Openstack::CloudManager::CloudTenant with 'id'=10000000000016 [WHERE "cloud_tenants"."type" IN ('ManageIQ::Providers::Openstack::CloudManager::CloudTenant')]
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:353:in `raise_record_not_found_exception!'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:479:in `find_one'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:458:in `find_with_ids'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:66:in `find'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/querying.rb:3:in `find'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/core.rb:151:in `find'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:448:in `block in reload'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/scoping/default.rb:35:in `block in unscoped'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation.rb:351:in `scoping'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/scoping/default.rb:35:in `unscoped'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:448:in `reload'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/attribute_methods/dirty.rb:37:in `reload'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/associations.rb:258:in `reload'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/autosave_association.rb:235:in `reload'
	from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/aggregations.rb:13:in `reload'
	from (irb):46
	from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/console.rb:65:in `start'
	from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/console_helper.rb:9:in `start'
	from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:78:in `console'
	from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

Looking for the CloudTenant by name also fails:

irb> CloudTenant.find_by(name: 'HR Intranet')
=> nil

There are errors in evm.log related to inventory refresh:

[----] E, [2019-01-03T10:17:25.734125 #13498:7c0f74] ERROR -- : Error when saving InventoryCollection:<Disk>, strategy: local_db_find_missing_references with strategy: local_db_find_missing_references, saver_strategy: default, targeted: true. Message: Referential integrity check violated for {:hardware_id=>nil, :dev
ice_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Op
enstack::CloudManager:10000000000006
[----] I, [2019-01-03T10:17:25.734387 #13498:7c0f74]  INFO -- : Exception in realtime_block :ems_refresh - Timings: {:collect_inventory_for_targets=>4.834689140319824, :parse_inventory=>18.67089557647705, :parse_targeted_inventory=>18.671165704727173, :save_inventory=>0.3510749340057373, :ems_refresh=>23.85715484619
1406}
[----] E, [2019-01-03T10:17:25.734561 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [OSP], id: [10000000000006] Refresh failed
[----] E, [2019-01-03T10:17:25.734771 #13498:7c0f74] ERROR -- : [RuntimeError]: Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"}
 of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2019-01-03T10:17:25.734873 #13498:7c0f74] ERROR -- : /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:280:in `block in assert_referential_integrity'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:271:in `each'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:271:in `assert_referential_integrity'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:176:in `block (2 levels) in save!'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:173:in `each'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:173:in `block in save!'
/opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in `transaction'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:172:in `save!'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:84:in `save_inventory_collection!'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/base.rb:37:in `save_inventory'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/base.rb:24:in `save_inventory_object_inventory'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:29:in `block (2 levels) in save_collections'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:28:in `each'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:28:in `block in save_collections'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `each'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `each_with_index'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `save_collections'
/opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_inventory.rb:24:in `save_inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory/persister.rb:29:in `persist!'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:189:in `save_inventory'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-openstack-b2c9895b69f9/app/models/manageiq/providers/openstack/cloud_manager/refresher.rb:4:in `save_inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:109:in `block in refresh_targets_for_ems'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:109:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block (2 levels) in refresh'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block in refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:103:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:275:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:241:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'
[----] E, [2019-01-03T10:17:25.734975 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [OSP], id: [10000000000006] Unable to perform refresh for the following targets:
[----] E, [2019-01-03T10:17:25.735143 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh)  --- InventoryRefresh::TargetCollection [Collection of 24 targets] id [Collection of targets with id: [{:ems_ref=>"9bdc9aea-27b4-4ffb-b549-7401e6baecce"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}, {:ems_ref=>"1b39625c-221e-4aad-ace3-a09685ac2634"}, {:ems_ref=>"bfeeeac1-9ceb-4ec9-936b-0bb3825bdc21"}, {:ems_ref=>"brett-conversion"}, {:ems_ref=>"819468a7-c8de-47c9-b8a2-8d620bb77e1d"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"nova"}, {:ems_ref=>"brett-conversion"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"9d82b93c-6ebc-4223-ba72-cbb83800515d"}, {:ems_ref=>"819468a7-c8de-47c9-b8a2-8d620bb77e1d"}, {:ems_ref=>"bfeeeac1-9ceb-4ec9-936b-0bb3825bdc21"}, {:ems_ref=>"1b39625c-221e-4aad-ace3-a09685ac2634"}, {:ems_ref=>"e5d41219-2868-4f2e-a7d3-6da132048851"}, {:ems_ref=>"default"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}, {:ems_ref=>"39f6ca6e7cdd40278d67394d8144c2e7"}, {:ems_ref=>"39f6ca6e7cdd40278d67394d8144c2e7"}, {:ems_ref=>"77e1f734-385b-4552-a490-feb43d687457"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}]]
[----] I, [2019-01-03T10:17:25.779557 #13498:7c0f74]  INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2019-01-03T10:17:25.779831 #13498:7c0f74] ERROR -- : MIQ(MiqQueue#deliver) Message id: [10000000381207], Error: [Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006]
[----] E, [2019-01-03T10:17:25.780022 #13498:7c0f74] ERROR -- : [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2019-01-03T10:17:25.780214 #13498:7c0f74] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:103:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:275:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:241:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

After a full refresh of the provider, the CloudTenant named 'HR Intranet' is back in the inventory, but with a different id:

irb> ExtManagementSystem.find(10000000000006,).refresh_ems
=> nil
irb> CloudTenant.find_by(name: 'HR Intranet')
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000018, name: "HR Intranet", description: "", enabled: true, ems_ref: "39f6ca6e7cdd40278d67394d8144c2e7", ems_id: 10000000000006, created_at: "2019-01-03 15:50:40", updated_at: "2019-01-03 15:50:40", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

I have destroyed the plan and mapping, and cleaned up the OpenStack environment so that you can run it on your own.
Please reach out if you need help on the migration process.

Comment 5 Adam Grare 2019-01-07 16:14:06 UTC
Also needed to fix the graph integrity exceptions:
https://github.com/ManageIQ/manageiq/pull/18328
https://github.com/ManageIQ/manageiq-providers-openstack/pull/423

Comment 8 Yadnyawalk Tale 2019-01-11 20:11:51 UTC
Not fixed! I can confirm added code do exists on 5.10.0.31 appliance, but still reproduce inventory refresh issue with OSP.
Marking it FailedQA.

Comment 17 Yadnyawalk Tale 2019-01-15 10:04:47 UTC
Fixed! Verified on 5.10.0.31.20190108221820_a0968c8.

Before migration
=================
irb(main):003:0> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster'))
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

irb(main):004:0> CloudTenant.find(10000000000005)
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

irb(main):005:0> CloudTenant.find_by(name: 'qe_destination_1')
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>


After migration
================
irb(main):014:0> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster'))
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

irb(main):015:0> CloudTenant.find(10000000000005)
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>

irb(main):016:0> CloudTenant.find_by(name: 'qe_destination_1')
=> #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>


Please note we have tested it with respect to cloud tenant problem mentioned here.
We can still reproduce refresh inventory problem and we are going raise new issue for that.
@Ladislav and I tried to reproduce it on call and looks like current problem with refresh inventory is not a regression and kind of different thing than we had last time.

Thanks,

Comment 18 Ladislav Smola 2019-01-15 10:23:07 UTC
BZ with the new issue found https://bugzilla.redhat.com/show_bug.cgi?id=1666238


Note You need to log in before you can comment on or make changes to this bug.