Description of problem: The Swift Manager refresh fails. Unable to collect swift containers and objects. Version-Release number of selected component (if applicable): master How reproducible: always Steps to Reproduce: 1. Add a OpenStack cloud provider. 2. Navigate to Storage Managers > Swift Manager 3. Error is displayed in Last Refresh: "undefined method `each' for nil:NilClass" Actual results: Refresh fails Expected results: Refresh should succeed. Additional info: From log files [----] I, [2017-01-12T16:02:45.884075 #1371:3fd98c] INFO -- : MIQ(ManageIQ::Providers::StorageManager::SwiftManager::RefreshParser.ems_inv_to_hashes) Collecting data for EMS name: [overcloud Swift Manager] id: [2]... ==> evm.log <== [----] E, [2017-01-12T16:02:45.886951 #1371:3fd98c] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected([200, 204]) <=> Actual(404 Not Found) excon.error.response :body => "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\"/>\n<title>Error 404 Not Found</title>\n</head>\n<body><h2>HTTP ERROR 404</h2>\n<p>Problem accessing /v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/. Reason:\n<pre> Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n\n</body>\n</html>\n" :cookies => [ ] :headers => { "Cache-Control" => "must-revalidate,no-cache,no-store" "Content-Length" => "1406" "Content-Type" => "text/html;charset=ISO-8859-1" "Server" => "Jetty(8.1.17.v20150415)" "X-Content-Type-Options" => "nosniff" } :host => "192.0.2.10" :local_address => "10.0.1.5" :local_port => 60306 :path => "/v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/" :port => 8080 :reason_phrase => "Not Found" :remote_ip => "192.0.2.10" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" > ==> fog.log <== [----] E, [2017-01-12T16:02:45.887050 #1371:3fd98c] ERROR -- : excon.error #<Excon::Error::NotFound: Expected([200, 204]) <=> Actual(404 Not Found) excon.error.response :body => "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\"/>\n<title>Error 404 Not Found</title>\n</head>\n<body><h2>HTTP ERROR 404</h2>\n<p>Problem accessing /v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/. Reason:\n<pre> Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n\n</body>\n</html>\n" :cookies => [ ] :headers => { "Cache-Control" => "must-revalidate,no-cache,no-store" "Content-Length" => "1406" "Content-Type" => "text/html;charset=ISO-8859-1" "Server" => "Jetty(8.1.17.v20150415)" "X-Content-Type-Options" => "nosniff" } :host => "192.0.2.10" :local_address => "10.0.1.5" :local_port => 60306 :path => "/v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/" :port => 8080 :reason_phrase => "Not Found" :remote_ip => "192.0.2.10" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" > [----] W, [2017-01-12T16:02:45.887302 #1371:3fd98c] WARN -- : MIQ(OpenstackHandle::Handle.accessor_for_accessible_tenants) HTTP 404 Error during OpenStack request. Skipping inventory item Storage #<Proc:0x0000000b90b928@/home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/openstack/openstack_handle/multi_tenancy/loop.rb:7 (lambda)> Expected([200, 204]) <=> Actual(404 Not Found) excon.error.response :body => "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\"/>\n<title>Error 404 Not Found</title>\n</head>\n<body><h2>HTTP ERROR 404</h2>\n<p>Problem accessing /v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/. Reason:\n<pre> Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n<br/> \n\n</body>\n</html>\n" :cookies => [ ] :headers => { "Cache-Control" => "must-revalidate,no-cache,no-store" "Content-Length" => "1406" "Content-Type" => "text/html;charset=ISO-8859-1" "Server" => "Jetty(8.1.17.v20150415)" "X-Content-Type-Options" => "nosniff" } :host => "192.0.2.10" :local_address => "10.0.1.5" :local_port => 60306 :path => "/v1/AUTH_90533982c5eb4c17aa409e1214a98bd7/" :port => 8080 :reason_phrase => "Not Found" :remote_ip => "192.0.2.10" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" [----] I, [2017-01-12T16:02:45.887358 #1371:3fd98c] INFO -- : MIQ(ManageIQ::Providers::StorageManager::SwiftManager::RefreshParser.ems_inv_to_hashes) Collecting data for EMS name: [overcloud Swift Manager] id: [2]...Complete ==> development.log <== [----] D, [2017-01-12T16:02:45.888989 #1371:3fd98c] DEBUG -- : ManageIQ::Providers::StorageManager::SwiftManager Load (0.3ms) SELECT "ext_management_systems".* FROM "ext_management_systems" WHERE "ext_management_systems"."type" IN ('ManageIQ::Providers::StorageManager::SwiftManager') AND "ext_management_systems"."parent_ems_id" = $1 LIMIT $2 [["parent_ems_id", 3], ["LIMIT", 1]] [----] D, [2017-01-12T16:02:45.889435 #1371:3fd98c] DEBUG -- : ManageIQ::Providers::StorageManager::SwiftManager Inst Including Associations (0.2ms - 1rows) ==> evm.log <== [----] E, [2017-01-12T16:02:45.890888 #1371:3fd98c] ERROR -- : MIQ(ManageIQ::Providers::StorageManager::SwiftManager::Refresher#refresh) EMS: [overcloud Swift Manager], id: [2] Refresh failed [----] E, [2017-01-12T16:02:45.891033 #1371:3fd98c] ERROR -- : [NoMethodError]: undefined method `each' for nil:NilClass Method:[rescue in block in refresh] [----] E, [2017-01-12T16:02:45.891174 #1371:3fd98c] ERROR -- : /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/storage_manager/swift_manager/refresh_parser/cross_linkers/openstack.rb:14:in `cross_link' /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/storage_manager/swift_manager/refresh_parser/cross_linkers.rb:13:in `cross_link' /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/storage_manager/swift_manager/refresh_parser.rb:30:in `ems_inv_to_hashes' /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/storage_manager/swift_manager/refresh_parser.rb:9:in `ems_inv_to_hashes' /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/storage_manager/swift_manager/refresher.rb:6:in `parse_legacy_inventory' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:122:in `block in parse_targeted_inventory' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:122:in `parse_targeted_inventory' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each' /home/rwsu/github/temp/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh' /home/rwsu/github/temp/manageiq/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh' /home/rwsu/github/temp/manageiq/app/models/ems_refresh.rb:92:in `block in refresh' /home/rwsu/github/temp/manageiq/app/models/ems_refresh.rb:91:in `each' /home/rwsu/github/temp/manageiq/app/models/ems_refresh.rb:91:in `refresh' /home/rwsu/github/temp/manageiq/app/models/miq_queue.rb:347:in `block in deliver' /home/rwsu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:89:in `block in timeout' /home/rwsu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:34:in `block in catch' /home/rwsu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:34:in `catch' /home/rwsu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:34:in `catch' /home/rwsu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:104:in `timeout' /home/rwsu/github/temp/manageiq/app/models/miq_queue.rb:343:in `deliver' /home/rwsu/github/temp/manageiq/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message' /home/rwsu/github/temp/manageiq/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message' /home/rwsu/github/temp/manageiq/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work' /home/rwsu/github/temp/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `loop' /home/rwsu/github/temp/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `do_work' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:334:in `block in do_work_loop' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:331:in `loop' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:331:in `do_work_loop' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:153:in `run' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:128:in `start' /home/rwsu/github/temp/manageiq/app/models/miq_worker/runner.rb:21:in `start_worker' /home/rwsu/github/temp/manageiq/app/models/miq_worker.rb:343:in `block in start' /home/rwsu/.rvm/gems/ruby-2.2.2/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /home/rwsu/.rvm/gems/ruby-2.2.2/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /home/rwsu/github/temp/manageiq/app/models/miq_worker.rb:341:in `start' /home/rwsu/github/temp/manageiq/app/models/miq_worker.rb:270:in `start_worker' /home/rwsu/github/temp/manageiq/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems' /home/rwsu/github/temp/manageiq/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers' /home/rwsu/github/temp/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `each' /home/rwsu/github/temp/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers' /home/rwsu/github/temp/manageiq/app/models/miq_server/worker_management/monitor.rb:52:in `block in sync_workers' /home/rwsu/github/temp/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `each' /home/rwsu/github/temp/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /home/rwsu/github/temp/manageiq/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:346:in `block in monitor' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:346:in `monitor' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:368:in `block (2 levels) in monitor_loop' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /home/rwsu/.rvm/gems/ruby-2.2.2/bundler/gems/manageiq-gems-pending-732f36463f96/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:368:in `block in monitor_loop' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:367:in `loop' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:367:in `monitor_loop' /home/rwsu/github/temp/manageiq/app/models/miq_server.rb:250:in `start' /home/rwsu/github/temp/manageiq/lib/workers/evm_server.rb:65:in `start' /home/rwsu/github/temp/manageiq/lib/workers/evm_server.rb:92:in `start' /home/rwsu/github/temp/manageiq/lib/workers/bin/evm_server.rb:4:in `<main>' [----] E, [2017-01-12T16:02:45.891231 #1371:3fd98c] ERROR -- : MIQ(ManageIQ::Providers::StorageManager::SwiftManager::Refresher#refresh) EMS: [overcloud Swift Manager], id: [2] Unable to perform refresh for the following targets: [----] E, [2017-01-12T16:02:45.891296 #1371:3fd98c] ERROR -- : MIQ(ManageIQ::Providers::StorageManager::SwiftManager::Refresher#refresh) --- ManageIQ::Providers::StorageManager::SwiftManager [overcloud Swift Manager] id [2]
Customer is on OSP 10 (14.0.2).
Richard, can you provider connectivity and credentials for the appliance in question? Thanks.
Jerry, I don't have a public server to share that exhibits the problem. Are you having trouble reproducing?
Correct.
*** Bug 1439285 has been marked as a duplicate of this bug. ***
It looks like the error happens here: https://github.com/ManageIQ/manageiq/blob/master/app/models/manageiq/providers/storage_manager/swift_manager/refresh_parser/cross_linkers/openstack.rb#L14 Given that, is it possible this error happens when swift exists, but no containers exist within swift (or no containers that the admin user can see)?
Tzu-Mainn, I verified this by creating container, and then uploaded file to that container and I'm no longer seen that error.
This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1441144 and will be closed. *** This bug has been marked as a duplicate of bug 1441144 ***