Description of problem: Every EMS full refresh we seen compute.instance.exists event type, and its error with [undefined method `key?' for nil:NilClass] [----] I, [2017-04-27T04:00:35.319721 #19396:5fd14c] INFO -- : MIQ(MiqEventHandler::Runner#get_message_via_drb) Message id: [1000012045911], MiqWorker id: [1000000600173], Zone: [Openstack Workers], Role: [event], Server: [], Id ent: [ems], Target id: [1000000000029], Instance id: [], Task id: [], Command: [EmsEvent.add_openstack], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [{:content=>{"oslo.message"=>"{\"_context _domain\": null, \"_context_roles\": [], \"_context_quota_class\": null, \"event_type\": \"compute.instance.exists\", \"_context_request_id\": \"req-4ed60751-f4d0-41fa-8ab8-c6a8d1061728\", \"_context_service_catalog\": [], \"timestamp\": \"2017-04-27 08:00:02.278315\", \"_context_user\": null, \"_unique_id\": \"452d0a0ff925481c86cf54e18e84db3d\", \"_context_resource_uuid\": null, \"_context_instance_lock_checked\": false, \"_context_is_admin_project\": true, \"_context_user_id\": null, \"payload\": {\"state_description\": \"\", \"availability_zone\": null, \"terminated_at\": \"\", \"ephemeral_gb\": 0, \"instance_type_id\": 11, \"bandwidth\": {}, \"deleted_at\": \"\", \"reservation_id\": \"r-a6s0lcqj\", \"instance_id\": \"4159b7d2-a2e9-46d8-8a4f-9f0b4e814906\", \"display_name\": \"lvsn02v910app004\", \"hostname\": \"lvsn02v910app004\", \"state\": \"suspended\", \"progress\": \"\", \"launched_at\": \"2017-03-28T19:00:39.000000\", \"metadata\": {}, \"node\": \"lvsnprod-compute-7.osp.gspt.net\", \"ramdisk_id\": \"\", \"access_ip_v6\": null, \"disk_gb\": 40, \"access_ip_v4\": null, \"kernel_id\": \"\", \"host\": \"lvsnprod-compute-7.osp.gspt.net\", \"user_id\": \"3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e\", \"image_ref_url\": \"http://10.49.134.201:9292/images/f650b0c9-884a-4d73-843f-c8cf163ed79a\", \"cell_name\": \"\", \"audit_period_beginning\": \"2017-04-27 07:00:00\", \"root_gb\": 40, \"tenant_id\": \"9fa05aef1a314b0d9b07b4427745865a\", \"created_at\": \"2017-03-28 18:59:32+00:00\", \"memory_mb\": 8192, \"instance_type\": \"V9_10-Flavor\", \"vcpus\": 6, \"image_meta\": {\"min_disk\": \"40\", \"container_format\": \"bare\", \"min_ram\": \"0\", \"disk_format\": \"qcow2\", \"base_image_ref\": \"f650b0c9-884a-4d73-843f-c8cf163ed79a\"}, \"architecture\": null, \"audit_period_ending\": \"2017-04-27 08:00:00\", \"os_type\": null, \"instance_flavor_id\": \"4e77b3cf-7ef6-4260-a4ac-9f09d8b76f6c\"}, \"_context_project_name\": null, \"_context_read_deleted\": \"no\", \"_context_user_identity\": \"- - - - -\", \"_context_auth_token\": null, \"_context_show_deleted\": false, \"_context_tenant\": null, \"priority\": \"INFO\", \"_context_read_only\": false, \"_context_is_admin\": true, \"_context_project_id\": null, \"_context_project_domain\": null, \"_context_timestamp\": \"2017-04-27T08:00:02.207887\", \"_context_user_domain\": null, \"_context_user_name\": null, \"publisher_id\": \"compute.lvsnprod-compute-7.osp.gspt.net\", \"message_id\": \"177fb337-230b-489e-ad97-7eb370c879de\", \"_context_remote_address\": null}", "oslo.version"=>"2.0"}, :context=>{}, :user_id=>nil, :priority=>nil, :content_type=>nil}], Dequeued in: [1.699107336] seconds [----] I, [2017-04-27T04:00:35.319900 #19396:5fd14c] INFO -- : MIQ(MiqQueue#deliver) Message id: [1000012045911], Delivering... [----] E, [2017-04-27T04:00:35.331089 #19396:5fd14c] ERROR -- : MIQ(MiqQueue#deliver) Message id: [1000012045911], Error: [undefined method `key?' for nil:NilClass] [----] E, [2017-04-27T04:00:35.331316 #19396:5fd14c] ERROR -- : [NoMethodError]: undefined method `key?' for nil:NilClass Method:[rescue in deliver] [----] E, [2017-04-27T04:00:35.331500 #19396:5fd14c] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/event_parser.rb:19:in `event_to_hash' /var/www/miq/vmdb/app/models/ems_event.rb:72:in `add_openstack' /var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:343:in `block in start' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:341:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:270:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `times' /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:52:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /var/www/miq/vmdb/app/models/miq_server.rb:346:in `block in monitor' /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:346:in `monitor' /var/www/miq/vmdb/app/models/miq_server.rb:368:in `block (2 levels) in monitor_loop' /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:250:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' ==== ceilometer has been stopped, and we using rabbitmq for the events. Version-Release number of selected component (if applicable): 5.7.1.3-20170221135006_818f133 OpenStack 10 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
*** Bug 1505401 has been marked as a duplicate of this bug. ***
Verified on RHOS 11 and 5.9.0.21