Bug 1488395

Summary:	Openstack::NetworkManager Refresh failed [NoMethodError]: undefined method `[]='
Product:	Red Hat CloudForms Management Engine	Reporter:	ncatling
Component:	Providers	Assignee:	Sam Lucidi <slucidi>
Status:	CLOSED ERRATA	QA Contact:	Ola Pavlenko <opavlenk>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	5.8.0	CC:	ademicev, gblomqui, jfrey, jhajyahy, jhardy, obarenbo, simaishi, tzumainn
Target Milestone:	GA	Keywords:	ZStream
Target Release:	5.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	5.9.0.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1564264 (view as bug list)		Environment:
Last Closed:	2018-03-01 13:16:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	Bug
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	Openstack	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1564264

Description ncatling 2017-09-05 09:28:23 UTC

Description of problem:

Intermittent refresh failure of Openstack::NetworkManager provider with [NoMethodError]: undefined method `[]='

ERROR -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::Refresher#refresh) EMS: [OpenStack OverCloud POD1 Network Manager], id: [1000000000006] Refresh failed
ERROR -- : [NoMethodError]: undefined method `[]=' for nil:NilClass  Method:[rescue in block in refresh]
ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:208:in `find_device_object'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:224:in `parse_network_port'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:110:in `block in get_network_ports'


Version-Release number of selected component (if applicable):
5.8.1.5


How reproducible:
Intermittent but seems to be related to the network port discovery.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

[----] E, [2017-09-05T03:49:36.112260 #37347:4d1138] ERROR -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::Refresher#refresh) EMS: [OpenStack OverCloud POD1 Network Manager], id: [1000000000006] Refresh failed
[----] E, [2017-09-05T03:49:36.114134 #37347:4d1138] ERROR -- : [NoMethodError]: undefined method `[]=' for nil:NilClass  Method:[rescue in block in refresh]
[----] E, [2017-09-05T03:49:36.114355 #37347:4d1138] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:208:in `find_device_object'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:224:in `parse_network_port'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:110:in `block in get_network_ports'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:15:in `process_collection_item'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:9:in `block (2 levels) in process_collection'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:24:in `safe_call'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:9:in `block in process_collection'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:9:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/refresh_parser_common/helper_methods.rb:9:in `process_collection'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:110:in `get_network_ports'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:40:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresh_parser.rb:8:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/network_manager/refresher.rb:6:in `parse_legacy_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `block in parse_targeted_inventory'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `parse_targeted_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:340:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:337:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:337:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:160:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:134:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:339:in `block in start_runner'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:337:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:348:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:253:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'
[----] E, [2017-09-05T03:49:36.114463 #37347:4d1138] ERROR -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::Refresher#refresh) EMS: [OpenStack OverCloud POD1 Network Manager], id: [1000000000006] Unable to perform refresh for the following targets:
[----] E, [2017-09-05T03:49:36.114598 #37347:4d1138] ERROR -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::Refresher#refresh)  --- ManageIQ::Providers::Openstack::NetworkManager [OpenStack OverCloud POD1 Network Manager] id [1000000000006]
[----] I, [2017-09-05T03:49:36.150071 #37347:4d1138]  INFO -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2017-09-05T03:49:36.150337 #37347:4d1138] ERROR -- : MIQ(MiqQueue#deliver) Message id: [1000001373770], Error: [undefined method `[]=' for nil:NilClass]
[----] E, [2017-09-05T03:49:36.150483 #37347:4d1138] ERROR -- : [EmsRefresh::Refreshers::EmsRefresherMixin::PartialRefreshError]: undefined method `[]=' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2017-09-05T03:49:36.150611 #37347:4d1138] ERROR -- : /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:50:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:340:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:337:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:337:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:160:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:134:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:339:in `block in start_runner'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:337:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:348:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-8f7d63f441d5/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:253:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

Comment 2 Sam Lucidi 2017-09-05 18:16:17 UTC

This appears to already be addressed in the new refresh system, this PR should fix it in the old one: https://github.com/ManageIQ/manageiq-providers-openstack/pull/90

Comment 6 errata-xmlrpc 2018-03-01 13:16:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0380