Description of problem: After creating 100 tenants and selecting one of them to be deleted, it was not deleted. Also no flash message appeared. when having only 3 tenants, deleting one of them works fine Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create 100 cloud tenants from OSP 2. add _member_ role to admin for all created tenants 3. Wait until all tenants appear on CFME UI 4. Select a tenant and click Configuration > Remove cloud tenants Actual results: Tenants was not deleted and no flash message was rendered upon deletion. Also error appeared on log: [----] E, [2018-02-04T03:59:12.068260 #16331:101b138] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#raw_delete_cloud_tenant) tenant=[Tenant100], error: Expected([200, 204]) <=> Actual(403 Forbidden) Expected results: Flash message should appear Tenant should be deleted from UI and OSP Additional info: [root@ibm-p8-01-fsp log]# grep -i Tenant100 evm.log [----] I, [2018-02-04T03:46:15.418571 #32425:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#sync_tenants) CloudTenant Tenant100 has no tenant [----] I, [2018-02-04T03:46:15.418983 #32425:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#sync_tenants) Creating Tenant with parameters: {:name=>"Tenant100", :description=>"Tenant100", :source=>#<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 1000000000094, name: "Tenant100", description: nil, enabled: true, ems_ref: "029d8201c9a6496493851af70ee6e83d", ems_id: 1000000000005, created_at: "2018-02-04 08:46:08", updated_at: "2018-02-04 08:46:08", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>} [----] I, [2018-02-04T03:46:15.440260 #32425:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#sync_tenants) New Tenant Tenant100 created [----] I, [2018-02-04T03:46:15.443891 #32425:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#sync_tenants) CloudTenant Tenant100 saved [----] I, [2018-02-04T03:59:05.012930 #16415:17c3e80] INFO -- : <AuditSuccess> MIQ(CloudTenantController.block in process_cloud_tenants) userid: [admin] - [Tenant100] Record delete initiated [----] E, [2018-02-04T03:59:12.068260 #16331:101b138] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#raw_delete_cloud_tenant) tenant=[Tenant100], error: Expected([200, 204]) <=> Actual(403 Forbidden) [----] I, [2018-02-04T04:03:52.739573 #25568:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#update_source_tenant) CloudTenant Tenant100 has tenant Tenant100 [----] I, [2018-02-04T04:03:52.740083 #25568:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#update_source_tenant) Updating Tenant Tenant100 with parameters: {:name=>"Tenant100", :description=>"Tenant100", :source=>#<ManageIQ::Providers::Openstack::CloudManager::CloudTenant id: 1000000000094, name: "Tenant100", description: nil, enabled: true, ems_ref: "029d8201c9a6496493851af70ee6e83d", ems_id: 1000000000005, created_at: "2018-02-04 08:46:08", updated_at: "2018-02-04 08:46:08", type: "ManageIQ::Providers::Openstack::CloudManager::Clou...", parent_id: nil>} [----] I, [2018-02-04T04:03:52.755809 #25568:101b138] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#sync_tenants) CloudTenant Tenant100 saved
Tenant CRUD operations does not work on RHOS 9, After creating 100 tenants from OSP side. See error on evm log: [----] I, [2018-02-05T06:47:50.357106 #16432:198b81c] INFO -- : MIQ(MiqQueue.put) Message id: [1000000023482], id: [], Zone: [default], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::CloudTenant.create_cloud_tenant], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [1000000000005, {:name=>"new_tenant"}] [----] I, [2018-02-05T06:47:55.214888 #16339:fff140] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [1000000023482], MiqWorker id: [1000000000003], Zone: [default], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::CloudTenant.create_cloud_tenant], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [1000000000005, {:name=>"new_tenant"}], Dequeued in: [4.873126789] seconds [----] E, [2018-02-05T06:47:56.413600 #16339:fff140] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant.raw_create_cloud_tenant) tenant=[new_tenant], error: Expected([200]) <=> Actual(403 Forbidden) [----] E, [2018-02-05T06:47:57.259499 #16432:198b574] ERROR -- : MIQ(cloud_tenant_controller-wait_for_task): Unable to create Cloud Tenant "new_tenant": Expected([200]) <=> Actual(403 Forbidden) [----] E, [2018-02-05T07:57:01.155683 #16339:fff140] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#raw_update_cloud_tenant) tenant=[Tenant10], error: Expected([200, 204]) <=> Actual(403 Forbidden) [----] E, [2018-02-05T07:57:04.288444 #16432:198b574] ERROR -- : MIQ(cloud_tenant_controller-wait_for_task): Unable to update Cloud Tenant "Tenant10_updated": Expected([200, 204]) <=> Actual(403 Forbidden) [----] I, [2018-02-05T08:15:25.621080 #15731:fff140] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::CloudTenant#update_source_tenant) CloudTenant Tenant10 has tenant Tenant10
I can reproduce the lack of flash message and have found the source of that problem, but I can't reproduce the inability to delete cloud tenants even with 100+ of them. Can you provide some more context about the circumstances under which the 403 happens? The fog.log showing the exact error would also be helpful.
the bug happens on RHOS9 and RHOS12
Fixes for Fine https://github.com/ManageIQ/manageiq/pull/16973 https://github.com/ManageIQ/manageiq-gems-pending/pull/337#issuecomment-364148730 https://github.com/ManageIQ/manageiq-ui-classic/pull/3386 Fixes for master/Gaprindashvili https://github.com/ManageIQ/manageiq-providers-openstack/pull/225
Verified on 5.10.0.2