Opening this program to track an issue found while triaging https://bugzilla.redhat.com/show_bug.cgi?id=1712049 ====== Now, we know that openstack events are being processed by Cloudforms, I tried reproducing customer scenario in my lab environment. My Env Details: ~~~ CFME v5.9.9.3 RHOSP 10 ~~~ Scenario 1 : Creating Floating IP using openstack CLI and verify CFME targeted refresh I created floating ip with openstack CLI: ~~~ [stack@instack ~]$ openstack floating ip create public +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2019-07-02T06:04:48Z | | description | | | fixed_ip_address | None | | floating_ip_address | 10.0.0.211 | | floating_network_id | ccaa9362-17aa-4104-8fa4-953db56cf8ab | | headers | | | id | b1484031-eeff-4679-9f01-9d80348fbb32 | | port_id | None | | project_id | e18b52b58c90410ab8d383a87d138fc2 | | project_id | e18b52b58c90410ab8d383a87d138fc2 | | revision_number | 1 | | router_id | None | | status | DOWN | | updated_at | 2019-07-02T06:04:48Z | +---------------------+--------------------------------------+ [stack@instack ~]$ openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | +--------------------------------------+---------------------+------------------+--------------------------------------+ | 513fc84e-8d2f-48cf-9234-9b85f05f14c9 | 10.0.0.206 | None | None | | b1484031-eeff-4679-9f01-9d80348fbb32 | 10.0.0.211 | None | None | | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 172.22.26.8 | 6fb26828-b392-4282-ac9f-fd3fa0f94c93 | +--------------------------------------+---------------------+------------------+--------------------------------------+ ~~~ CFME captured the event: ~~~ [----] I, [2019-07-02T06:03:48.083711 #14647:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::EventCatcher::Runner#queue_event) EMS [192.0.2.13] as [admin] Caught event [floatingip.create.start] ~~~ Targeted refresh logs was observed for floaing ip 10.0.0.211 (b1484031-eeff-4679-9f01-9d80348fbb32): ~~~ [----] I, [2019-07-02T06:03:59.203029 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) Refreshing all targets... [----] I, [2019-07-02T06:03:59.203245 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [openstack], id: [1000000000003] Refreshing targets for EMS... [----] I, [2019-07-02T06:03:59.203408 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [openstack], id: [1000000000003] ManagerRefresh::TargetCollection [Collection of 2 targets] id [Collection of targets with id: [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"b1484031-eeff-4679-9f01-9d80348fbb32"}]] [----] I, [2019-07-02T06:03:59.203593 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#collect_inventory_for_targets) Filtering inventory for ManagerRefresh::TargetCollection [Collection of 2 targets] id: [Collection of targets with id: [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"b1484031-eeff-4679-9f01-9d80348fbb32"}]]... [----] I, [2019-07-02T06:04:00.015655 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#collect_inventory_for_targets) Filtering inventory...Complete [----] I, [2019-07-02T06:04:00.015915 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh_targets_for_ems) EMS: [openstack], id: [1000000000003] Refreshing target ManagerRefresh::TargetCollection [Collection of 2 targets] id [Collection of targets with id: [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"b1484031-eeff-4679-9f01-9d80348fbb32"}]]... [----] I, [2019-07-02T06:04:00.298248 #14647:91a67c] INFO -- : Querying OpenStack for events newer than 2019-07-02T06:04:33+00:00... [----] I, [2019-07-02T06:04:00.451359 #14656:8f01b0] INFO -- : Querying OpenStack for events newer than 2019-07-02T06:04:33+00:00... [----] I, [2019-07-02T06:04:02.044613 #24508:451104] INFO -- : MIQ(ManagerRefresh::SaveInventory.save_inventory) EMS: [openstack], id: [1000000000003] Saving EMS Inventory... [----] I, [2019-07-02T06:04:02.160821 #24508:451104] INFO -- : MIQ(ManagerRefresh::SaveInventory.save_inventory) EMS: [openstack], id: [1000000000003] Saving EMS Inventory...Complete [----] I, [2019-07-02T06:04:02.161079 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh_targets_for_ems) EMS: [openstack], id: [1000000000003] Refreshing target ManagerRefresh::TargetCollection [Collection of 2 targets] id [Collection of targets with id: [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"b1484031-eeff-4679-9f01-9d80348fbb32"}]]...Complete [----] I, [2019-07-02T06:04:02.161184 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh_targets_for_ems) EMS: [openstack], id: [1000000000003] ManagerRefresh Post Processing ManagerRefresh::TargetCollection [Collection of 2 targets] id [Collection of targets with id: [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"b1484031-eeff-4679-9f01-9d80348fbb32"}]]... ~~~ Despite of the targeted refresh, floating ip inventory is not update into the VMDB database: ~~~ [root@beaker-cfme vmdb]# psql vmdb_production psql (9.5.14) Type "help" for help. vmdb_production=# select * from floating_ips; id | type | ems_ref | address | ems_id | vm_id | cloud_network_only | cloud_tenant_id | network_port_id | cloud_network_id | fixed_ip_address | status ---------------+------------------------------------------------------------+--------------------------------------+------------+---------------+---------------+--------------------+-----------------+-----------------+------------------+------------------+-------- 1000000000002 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 1000000000004 | 1000000000041 | | 1000000000001 | 1000000000011 | 1000000000003 | 172.22.26.8 | ACTIVE (1 row) ~~~ After a complete manual refresh, the information about floating ips were found in the database. ~~~ [root@beaker-cfme vmdb]# psql vmdb_production psql (9.5.14) Type "help" for help. vmdb_production=# select * from floating_ips; id | type | ems_ref | address | ems_id | vm_id | cloud_network_only | cloud_tenant_id | network_port_id | cloud_network_id | fixed_ip_address | status ---------------+------------------------------------------------------------+--------------------------------------+------------+---------------+---------------+--------------------+-----------------+-----------------+------------------+------------------+-------- 1000000000002 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 1000000000004 | 1000000000041 | | 1000000000001 | 1000000000011 | 1000000000003 | 172.22.26.8 | ACTIVE 1000000000003 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | 513fc84e-8d2f-48cf-9234-9b85f05f14c9 | 10.0.0.206 | 1000000000004 | | | 1000000000001 | | 1000000000003 | | DOWN 1000000000004 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | b1484031-eeff-4679-9f01-9d80348fbb32 | 10.0.0.211 | 1000000000004 | | | 1000000000001 | | 1000000000003 | | DOWN (3 rows) ~~~ Scenario 2 : Deleting openstack instance from openstack CLI / Deleting Openstack instance from CFME Web console This works great, events were processed and the instance was moved to <Archived> into the VMDB database. Scenario 3 : Attach floating ip to an instance using openstack CLI - Information before attaching the floating ip to instance: ~~~ [stack@instack ~]$ openstack server list +--------------------------------------+------------------+---------+-------------------------------+------------+ | ID | Name | Status | Networks | Image Name | +--------------------------------------+------------------+---------+-------------------------------+------------+ | c7134235-23a4-4914-9a87-099209fa33ed | rahul-test | ACTIVE | local=172.22.26.15 | cirros | | 8d53192d-93ba-432a-8b35-12eec3dee5bc | demo_inst | ACTIVE | local=172.22.26.8, 10.0.0.209 | cirros | | d526f7aa-3189-4c29-8b8e-19e985f86d38 | Instance-RHEL7.5 | SHUTOFF | local=172.22.26.7 | rhel7.5 | +--------------------------------------+------------------+---------+-------------------------------+------------+ [stack@instack ~]$ openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | +--------------------------------------+---------------------+------------------+--------------------------------------+ | 513fc84e-8d2f-48cf-9234-9b85f05f14c9 | 10.0.0.206 | None | None | | b1484031-eeff-4679-9f01-9d80348fbb32 | 10.0.0.211 | None | None | | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 172.22.26.8 | 6fb26828-b392-4282-ac9f-fd3fa0f94c93 | +--------------------------------------+---------------------+------------------+--------------------------------------+ [root@beaker-cfme vmdb]# psql vmdb_production psql (9.5.14) Type "help" for help. vmdb_production=# select * from floating_ips; id | type | ems_ref | address | ems_id | vm_id | cloud_network_only | cloud_tenant_id | network_port_id | cloud_network_id | fixed_ip_address | status ---------------+------------------------------------------------------------+--------------------------------------+------------+---------------+---------------+--------------------+-----------------+-----------------+------------------+------------------+-------- 1000000000002 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 1000000000004 | 1000000000041 | | 1000000000001 | 1000000000011 | 1000000000003 | 172.22.26.8 | ACTIVE 1000000000003 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | 513fc84e-8d2f-48cf-9234-9b85f05f14c9 | 10.0.0.206 | 1000000000004 | | | 1000000000001 | | 1000000000003 | | DOWN 1000000000005 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | b1484031-eeff-4679-9f01-9d80348fbb32 | 10.0.0.211 | 1000000000004 | | | 1000000000001 | | 1000000000003 | | DOWN (3 rows) ~~~ - Attaching floating ip 10.0.0.206 (513fc84e-8d2f-48cf-9234-9b85f05f14c9) to instance rahul-test (c7134235-23a4-4914-9a87-099209fa33ed) ~~~ [stack@instack ~]$ openstack server add floating ip c7134235-23a4-4914-9a87-099209fa33ed 10.0.0.206 [stack@instack ~]$ openstack server list +--------------------------------------+------------------+---------+--------------------------------+------------+ | ID | Name | Status | Networks | Image Name | +--------------------------------------+------------------+---------+--------------------------------+------------+ | c7134235-23a4-4914-9a87-099209fa33ed | rahul-test | ACTIVE | local=172.22.26.15, 10.0.0.206 | cirros | | 8d53192d-93ba-432a-8b35-12eec3dee5bc | demo_inst | ACTIVE | local=172.22.26.8, 10.0.0.209 | cirros | | d526f7aa-3189-4c29-8b8e-19e985f86d38 | Instance-RHEL7.5 | SHUTOFF | local=172.22.26.7 | rhel7.5 | +--------------------------------------+------------------+---------+--------------------------------+------------+ ~~~ - CFME caught the event ~~~ [----] I, [2019-07-02T07:03:36.502430 #14647:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::EventCatcher::Runner#queue_event) EMS [192.0.2.13] as [admin] Caught event [floatingip.update.start] ~~~ - Noticed EmsRefresh happening ~~~ [----] I, [2019-07-02T07:03:23.975033 #24508:451104] INFO -- : MIQ(MiqQueue.get) Message id: [1000000385053], MiqWorker id: [1000000001502], Zone: [default], Role: [ems_inventory], Server: [], Ident: [ems_1000000000003], Target id: [], Instance id: [], Task id: [], Comma nd: [EmsRefresh.refresh], Timeout: [7200], Priority: [100], State: [dequeue], Deliver On: [], Data: [2 bytes], Args: [], Dequeued in: [9.14187383] seconds [...] [----] I, [2019-07-02T07:03:23.988185 #24508:451104] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#collect_inventory_for_targets) Filtering inventory for ManagerRefresh::TargetCollection [Collection of 2 targets] id: [Collection of targets with id : [{:ems_ref=>"e18b52b58c90410ab8d383a87d138fc2"}, {:ems_ref=>"513fc84e-8d2f-48cf-9234-9b85f05f14c9"}]]... ~~~ - After targeted refresh, vmdb database does not show the floating ip (10.0.0.206) ~~~ vmdb_production=# select * from floating_ips; id | type | ems_ref | address | ems_id | vm_id | cloud_network_only | cloud_tenant_id | network_port_id | cloud_network_id | fixed_ip_address | status ---------------+------------------------------------------------------------+--------------------------------------+------------+---------------+---------------+--------------------+-----------------+-----------------+------------------+------------------+-------- 1000000000002 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | d1476653-dbfc-4451-b038-291ff204fbe3 | 10.0.0.209 | 1000000000004 | 1000000000041 | | 1000000000001 | 1000000000011 | 1000000000003 | 172.22.26.8 | ACTIVE 1000000000005 | ManageIQ::Providers::Openstack::NetworkManager::FloatingIp | b1484031-eeff-4679-9f01-9d80348fbb32 | 10.0.0.211 | 1000000000004 | | | 1000000000001 | | 1000000000003 | | DOWN (2 rows) ~~~ It seems that the targeted refresh is not working as expected for floating IP create and update events.
Marek, the targeted refresh is not working for security groups as well. I have tested and reproduced the issue.
Do we have results available from 5.10 testing?
I've been able to reproduce this now, and it looks to me like this is (part of) the same problem reported in https://bugzilla.redhat.com/show_bug.cgi?id=1596058. https://bugzilla.redhat.com/show_bug.cgi?id=1596058 is still ON_QA, but if that's successfully VERIFIED then backporting the fix to 5.9 should solve the problem. These are the PRs referenced in https://bugzilla.redhat.com/show_bug.cgi?id=1596058: https://github.com/ManageIQ/manageiq-providers-openstack/pull/323 https://github.com/ManageIQ/manageiq-content/pull/371 https://github.com/ManageIQ/manageiq-content/pull/372
*** This bug has been marked as a duplicate of bug 1596058 ***