Bug 1662126
| Summary: | CloudTenant is deleted by provider refresh | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat CloudForms Management Engine | Reporter: | Fabien Dupont <fdupont> | ||||
| Component: | Providers | Assignee: | Sam Lucidi <slucidi> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Yadnyawalk Tale <ytale> | ||||
| Severity: | high | Docs Contact: | Red Hat CloudForms Documentation <cloudforms-docs> | ||||
| Priority: | urgent | ||||||
| Version: | 5.10.0 | CC: | agrare, bthurber, dmetzger, jfrey, jhardy, kkulkarn, lsmola, maufart, miguel, obarenbo, simaishi, slucidi, ytale | ||||
| Target Milestone: | GA | ||||||
| Target Release: | 5.10.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | 5.10.0.31 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-02-12 16:52:40 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | Openstack | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1601090 | ||||||
| Attachments: |
|
||||||
I couldn't verify that targeted refresh would cause this behavior. Can we repeat the experiment, to find out if it's always reproducible? (I've switched to :debug on the testing appliance) I'm able to reproduce 100%. I've created a mapping and a plan, then run the migration, using the following commands in Rails console: irb> osp_mapping = TransformationMapping.create(:name => 'BZ1662126 Mapping') irb> source_vm = Vm.find_by(:name => "fdupont-test-migration", :vendor => 'vmware') irb> source_cluster = source_vm.ems_cluster irb> destination_cluster = CloudTenant.find_by(:name => 'HR Intranet') irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_cluster, :destination => destination_cluster) irb> source_network = source_vm.hardware.nics.first.lan irb> destination_network = CloudNetwork.find_by_name('external') irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_network, :destination => destination_network) irb> source_storage = source_vm.hardware.disks.select { |d| d.device_type == 'disk' }.first.storage irb> destination_storage = CloudVolumeType.find_by(name: 'default') irb> osp_mapping.transformation_mapping_items << TransformationMappingItem.new(:source => source_storage, :destination => destination_storage) irb> destination_flavor = Flavor.find_by(name: 'x1.small') irb> destination_security_group = SecurityGroup.find_by(name: 'default', cloud_tenant_id: destination_cluster.id) irb> osp_plan = ServiceTemplateTransformationPlan.create_catalog_item(:name => "BZ1662126 Plan", :config_info => { :transformation_mapping_id => osp_mapping.id, :actions => [ { :vm_id => source_vm.id.to_s, :osp_flavor_id => destination_flavor.id, :osp_security_group_id => destination_security_group.id } ] }) irb> osp_plan.provision_request(User.first) Before the inventory refresh, I can get the destination_cluster (which is a CloudTenant) from the DB: irb> destination_cluster.reload => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000016, name: "HR Intranet", description: "", enabled: true, ems_ref: "39f6ca6e7cdd40278d67394d8144c2e7", ems_id: 10000000000006, created_at: "2018-12-26 09:25:55", updated_at: "2018-12-26 09:25:55", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> After the refresh, here is the error I get: irb> destination_cluster.reload ActiveRecord::RecordNotFound: Couldn't find ManageIQ::Providers::Openstack::CloudManager::CloudTenant with 'id'=10000000000016 [WHERE "cloud_tenants"."type" IN ('ManageIQ::Providers::Openstack::CloudManager::CloudTenant')] from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:353:in `raise_record_not_found_exception!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:479:in `find_one' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:458:in `find_with_ids' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation/finder_methods.rb:66:in `find' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/querying.rb:3:in `find' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/core.rb:151:in `find' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:448:in `block in reload' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/scoping/default.rb:35:in `block in unscoped' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/relation.rb:351:in `scoping' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/scoping/default.rb:35:in `unscoped' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:448:in `reload' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/attribute_methods/dirty.rb:37:in `reload' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/associations.rb:258:in `reload' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/autosave_association.rb:235:in `reload' from /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/aggregations.rb:13:in `reload' from (irb):46 from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/console.rb:65:in `start' from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/console_helper.rb:9:in `start' from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:78:in `console' from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:49:in `run_command!' from /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/commands.rb:18:in `<top (required)>' from bin/rails:4:in `require' from bin/rails:4:in `<main>' Looking for the CloudTenant by name also fails: irb> CloudTenant.find_by(name: 'HR Intranet') => nil There are errors in evm.log related to inventory refresh: [----] E, [2019-01-03T10:17:25.734125 #13498:7c0f74] ERROR -- : Error when saving InventoryCollection:<Disk>, strategy: local_db_find_missing_references with strategy: local_db_find_missing_references, saver_strategy: default, targeted: true. Message: Referential integrity check violated for {:hardware_id=>nil, :dev ice_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Op enstack::CloudManager:10000000000006 [----] I, [2019-01-03T10:17:25.734387 #13498:7c0f74] INFO -- : Exception in realtime_block :ems_refresh - Timings: {:collect_inventory_for_targets=>4.834689140319824, :parse_inventory=>18.67089557647705, :parse_targeted_inventory=>18.671165704727173, :save_inventory=>0.3510749340057373, :ems_refresh=>23.85715484619 1406} [----] E, [2019-01-03T10:17:25.734561 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [OSP], id: [10000000000006] Refresh failed [----] E, [2019-01-03T10:17:25.734771 #13498:7c0f74] ERROR -- : [RuntimeError]: Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006 Method:[block (2 levels) in <class:LogProxy>] [----] E, [2019-01-03T10:17:25.734873 #13498:7c0f74] ERROR -- : /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:280:in `block in assert_referential_integrity' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:271:in `each' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:271:in `assert_referential_integrity' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:176:in `block (2 levels) in save!' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:173:in `each' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:173:in `block in save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in `transaction' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:172:in `save!' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/saver/base.rb:84:in `save_inventory_collection!' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/base.rb:37:in `save_inventory' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/base.rb:24:in `save_inventory_object_inventory' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:29:in `block (2 levels) in save_collections' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:28:in `each' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:28:in `block in save_collections' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `each' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `each_with_index' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_collection/topological_sort.rb:26:in `save_collections' /opt/rh/cfme-gemset/gems/inventory_refresh-0.1.1/lib/inventory_refresh/save_inventory.rb:24:in `save_inventory' /var/www/miq/vmdb/app/models/manageiq/providers/inventory/persister.rb:29:in `persist!' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:189:in `save_inventory' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-openstack-b2c9895b69f9/app/models/manageiq/providers/openstack/cloud_manager/refresher.rb:4:in `save_inventory' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:109:in `block in refresh_targets_for_ems' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:109:in `refresh_targets_for_ems' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block (2 levels) in refresh' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block in refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `each' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:103:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `each' /var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `refresh' /var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method' /usr/share/ruby/timeout.rb:93:in `block in timeout' /usr/share/ruby/timeout.rb:33:in `block in catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:108:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method' /var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver' /var/www/miq/vmdb/app/models/user.rb:275:in `with_user_group' /var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork' /var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner' /var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:241:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' [----] E, [2019-01-03T10:17:25.734975 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [OSP], id: [10000000000006] Unable to perform refresh for the following targets: [----] E, [2019-01-03T10:17:25.735143 #13498:7c0f74] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) --- InventoryRefresh::TargetCollection [Collection of 24 targets] id [Collection of targets with id: [{:ems_ref=>"9bdc9aea-27b4-4ffb-b549-7401e6baecce"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}, {:ems_ref=>"1b39625c-221e-4aad-ace3-a09685ac2634"}, {:ems_ref=>"bfeeeac1-9ceb-4ec9-936b-0bb3825bdc21"}, {:ems_ref=>"brett-conversion"}, {:ems_ref=>"819468a7-c8de-47c9-b8a2-8d620bb77e1d"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"nova"}, {:ems_ref=>"brett-conversion"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"9d82b93c-6ebc-4223-ba72-cbb83800515d"}, {:ems_ref=>"819468a7-c8de-47c9-b8a2-8d620bb77e1d"}, {:ems_ref=>"bfeeeac1-9ceb-4ec9-936b-0bb3825bdc21"}, {:ems_ref=>"1b39625c-221e-4aad-ace3-a09685ac2634"}, {:ems_ref=>"e5d41219-2868-4f2e-a7d3-6da132048851"}, {:ems_ref=>"default"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}, {:ems_ref=>"39f6ca6e7cdd40278d67394d8144c2e7"}, {:ems_ref=>"39f6ca6e7cdd40278d67394d8144c2e7"}, {:ems_ref=>"77e1f734-385b-4552-a490-feb43d687457"}, {:ems_ref=>"641a64a1a42d429f9606b345f328d306"}, {:ems_ref=>"371f96c1-d3c5-4506-8a95-ffe8e1fc6276"}]] [----] I, [2019-01-03T10:17:25.779557 #13498:7c0f74] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) Refreshing all targets...Complete [----] E, [2019-01-03T10:17:25.779831 #13498:7c0f74] ERROR -- : MIQ(MiqQueue#deliver) Message id: [10000000381207], Error: [Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006] [----] E, [2019-01-03T10:17:25.780022 #13498:7c0f74] ERROR -- : [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: Referential integrity check violated for {:hardware_id=>nil, :device_name=>"Root disk", :location=>"vda", :size=>17179869184, :device_type=>"disk", :controller_type=>"openstack", :backing_id=>10000000000103, :backing_type=>"CloudVolume"} of InventoryCollection:<Disk>, strategy: local_db_find_missing_references because of missing foreign key hardware_id for ManageIQ::Providers::Openstack::CloudManager:10000000000006 Method:[block (2 levels) in <class:LogProxy>] [----] E, [2019-01-03T10:17:25.780214 #13498:7c0f74] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:103:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `each' /var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `refresh' /var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method' /usr/share/ruby/timeout.rb:93:in `block in timeout' /usr/share/ruby/timeout.rb:33:in `block in catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:108:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method' /var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver' /var/www/miq/vmdb/app/models/user.rb:275:in `with_user_group' /var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork' /var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner' /var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-078d1090482e/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:241:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' After a full refresh of the provider, the CloudTenant named 'HR Intranet' is back in the inventory, but with a different id: irb> ExtManagementSystem.find(10000000000006,).refresh_ems => nil irb> CloudTenant.find_by(name: 'HR Intranet') => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000018, name: "HR Intranet", description: "", enabled: true, ems_ref: "39f6ca6e7cdd40278d67394d8144c2e7", ems_id: 10000000000006, created_at: "2019-01-03 15:50:40", updated_at: "2019-01-03 15:50:40", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> I have destroyed the plan and mapping, and cleaned up the OpenStack environment so that you can run it on your own. Please reach out if you need help on the migration process. Also needed to fix the graph integrity exceptions: https://github.com/ManageIQ/manageiq/pull/18328 https://github.com/ManageIQ/manageiq-providers-openstack/pull/423 Not fixed! I can confirm added code do exists on 5.10.0.31 appliance, but still reproduce inventory refresh issue with OSP. Marking it FailedQA. Fixed! Verified on 5.10.0.31.20190108221820_a0968c8. Before migration ================= irb(main):003:0> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster')) => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> irb(main):004:0> CloudTenant.find(10000000000005) => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> irb(main):005:0> CloudTenant.find_by(name: 'qe_destination_1') => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> After migration ================ irb(main):014:0> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster')) => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> irb(main):015:0> CloudTenant.find(10000000000005) => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> irb(main):016:0> CloudTenant.find_by(name: 'qe_destination_1') => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000005, name: "qe_destination_1", description: "Project created by KK", enabled: true, ems_ref: "5247facd0f454778a0a1b1dae554efcf", ems_id: 10000000000006, created_at: "2019-01-10 05:56:30", updated_at: "2019-01-10 05:56:30", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> Please note we have tested it with respect to cloud tenant problem mentioned here. We can still reproduce refresh inventory problem and we are going raise new issue for that. @Ladislav and I tried to reproduce it on call and looks like current problem with refresh inventory is not a regression and kind of different thing than we had last time. Thanks, BZ with the new issue found https://bugzilla.redhat.com/show_bug.cgi?id=1666238 |
Created attachment 1516790 [details] Screenshot showing mangled mapping Description of problem: During VM Migration, the OpenStack provider is refreshed and this deletes the destination CloudTenant from the inventory. This leads to a failure of the migration. In the Migration UI, I have created a new mapping, named 'OSP Mapping', with 'Corp Website' as destination CloudTenant. Then, I've created a plan using the mapping. When running that plan, it fails with the following error (automation.log): [----] E, [2018-12-26T03:49:03.432217 #22761:d74f308] ERROR -- : Q-task_id([r10000000000031_service_template_transformation_plan_task_10000000000029]) Method STDERR: (drbunix:///tmp/automation_engine20181226-22761-1lj8znc) /var/www/miq/vmdb/app/models/service_template_transformation_plan_task.rb:70:in `destination_cluster': [fdupont-test-migration] Cluster #<EmsCluster:0x000000001ac7e060> has no mapping. (RuntimeError) And the error is 'normal', because the source cluster has no destination mapping: irb> TransformationMapping.find_by(name: 'OSP Mapping').destination(EmsCluster.find_by(name: 'V2V_Cluster')) => nil Looking at the mapping itself, I can see that there's still a reference to the destination tenant: irb> TransformationMapping.find_by(name: 'OSP Mapping').transformation_mapping_items.select { |tmi| tmi.source_type == 'EmsCluster' } => [#<TransformationMappingItem id: 10000000000037, source_id: 10000000000001, source_type: "EmsCluster", destination_id: 10000000000009, destination_type: "CloudTenant", transformation_mapping_id: 10000000000007, options: {}, created_at: "2018-12-26 09:17:59", updated_at: "2018-12-26 09:17:59">] But the CloudTenant with id 10000000000009 doesn't exist: irb> CloudTenant.find(10000000000009) ActiveRecord::RecordNotFound: Couldn't find CloudTenant with 'id'=10000000000009 Looking for the same tenant by name also returns nil: irb> CloudTenant.find_by(name: 'Corp Website') => nil Running a full refresh of the provider brings the CloudTenant back in the inventory, but with a different id: irb> CloudTenant.find_by(name: 'Corp Website') => #<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 10000000000017, name: "Corp Website", description: "", enabled: true, ems_ref: "ce5654dbd92d467ea135ca4feb9ccee8", ems_id: 10000000000006, created_at: "2018-12-26 09:46:12", updated_at: "2018-12-26 09:46:12", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil> So, it seems that the targeted inventory refresh deletes the original CloudTenant. In the UI, I can also see that the mapping is wrong: cf. attached screenshot.