Description of problem: Lots of error messages in /var/www/miq/vmdb/log/evm.log that look like this: [----] I, [2017-02-22T15:05:27.671517 #7129:69913c] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture) [realtime] Capture for ManageIQ::Providers::Openstack::CloudManager::Vm name: [test], id: [11000000000002]... [----] I, [2017-02-22T15:05:27.999618 #15587:879880] INFO -- : Querying Openstack Ceilometer for events newer than 2017-02-22T11:15:21.574284... [fog][WARNING] Unrecognized arguments: provider [----] E, [2017-02-22T15:05:28.861821 #7129:69913c] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "{\"code\": 404, \"description\": \"Resource 331efb38-2ca1-4330-99b7-d374baae5d5c does not exist\", \"title\": \"Not Found\"}" :cookies => [ ] :headers => { "Connection" => "close" "Content-Length" => "114" "Content-Type" => "application/json; charset=UTF-8" "Date" => "Wed, 22 Feb 2017 20:05:29 GMT" "Server" => "Apache/2.4.6 (Red Hat Enterprise Linux)" } :host => "myospems.exapmle.com" :local_address => "10.10.10.71" :local_port => 46082 :path => "/v1/resource/generic/331efb38-2ca1-4330-99b7-d374baae5d5c" :port => 13041 :reason_phrase => "Not Found" :remote_ip => "10.10.10.10" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" > [----] W, [2017-02-22T15:05:28.862158 #7129:69913c] WARN -- : OpenstackHandle::MetricDelegate list_meters filter:{"field"=>"resource_id", "value"=>"331efb38-2ca1-4330-99b7-d374baae5d5c"} Resource does not exist. A curl confirms what we are seeing in the logs: [root@cfme-01 vmdb]# curl -H 'X-Auth-Token:XXXXXXMYAUTHTOKEN' -X GET https://10.10.10.71:13041/v1/resource/generic/331efb38-2ca1-4330-99b7-d374baae5d5c -k <html> <head> <title>404 Not Found</title> </head> <body> <h1>404 Not Found</h1> The resource could not be found.<br /><br /> Resource a187e624-c475-4613-b33f-4eea973c281b does not exist </body> Version-Release number of selected component (if applicable): [root@vlp-cldfrmwkr02 vmdb]# rpm -qa | grep cfme cfme-5.7.0.17-1.el7cf.x86_64 cfme-gemset-5.7.0.17-1.el7cf.x86_64 cfme-appliance-5.7.0.17-1.el7cf.x86_64 rh-ruby23-rubygem-redhat_access_cfme-1.1.0-1.el7cf.noarch Connecting to OSP 9 How reproducible: 100% Steps to Reproduce: 1. Configure OSP Provider 2. Monitor evm.log for errors 3. Actual results: 404 errors in evm.log Expected results: No 404 errors in evm.log when running ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture method Additional info:
Mixed up the :host, :local_address, and :remote_ip when sanitizing customer logs.
In order to investigate this properly, is it possible to get an idea of how this error surfaces outside of the log - failed refresh or events capture... ?
The above logs seem to indicate that this happened during a performance capture (C&U collection possibly????), however, I am seeing much more of this in the logs now, which looks to be happening during an EMS Refresh: [----] I, [2017-03-07T04:01:44.786279 #3080:c9b134] INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) Refreshing all targets... [----] I, [2017-03-07T04:01:44.786398 #3080:c9b134] INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [vlp-ospdirect02], id: [11000000000003] Refreshing targets for EMS... [----] I, [2017-03-07T04:01:44.786502 #3080:c9b134] INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [vlp-ospdirect02], id: [11000000000003] ManageIQ::Providers::Openstack::InfraManager [direct02] id [11000000000003] [----] I, [2017-03-07T04:01:44.786623 #3080:c9b134] INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh_targets_for_ems) EMS: [direct02], id: [11000000000003] Refreshing target ManageIQ::Providers::Openstack::InfraManager [direct02] id [11000000000003]... [----] E, [2017-03-07T04:01:47.274424 #3072:c9b134] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "404 Not Found\n\nThe resource could not be found.\n\n " :cookies => [ ] :headers => { "Content-Length" => "52" "Content-Type" => "text/plain; charset=UTF-8" "Date" => "Tue, 07 Mar 2017 09:01:28 GMT" "X-Openstack-Request-Id" => "req-193ba00d-52b0-4c4f-9d4b-9ba5f486f87b" } :host => "10.10.10.5" :local_address => "10.10.10.71" :local_port => 52866 :path => "/v2.0/routers" :port => 9696 :reason_phrase => "Not Found" :remote_ip => "10.10.10.5" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" > [----] E, [2017-03-07T04:01:48.133105 #3072:c9b134] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "404 Not Found\n\nThe resource could not be found.\n\n " :cookies => [ ] :headers => { "Content-Length" => "52" "Content-Type" => "text/plain; charset=UTF-8" "Date" => "Tue, 07 Mar 2017 09:01:29 GMT" "X-Openstack-Request-Id" => "req-173cb267-b2b6-4df2-8aca-6badd01b285d" } :host => "10.10.10.5" :local_address => "10.10.10.71" :local_port => 52894 :path => "/v2.0/floatingips" :port => 9696 :reason_phrase => "Not Found" :remote_ip => "10.10.10.5" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" > [----] I, [2017-03-07T04:01:48.134196 #3072:c9b134] INFO -- : MIQ(EmsRefresh.save_ems_network_inventory) EMS: [direct02 Network Manager], id: [11000000000004] Saving EMS Network Inventory...
Dear customer, The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. If you have any concerns about this, please let us know. Thanks and regards!